Websocket connection through nginx reverse proxy is not working, probably due to Autobahn introduced in #426

[pablo-quilez]

Hi,

we have some trouble after updating rosbridge. We discovered that probably since Autobahn, nginx is not working as reverse proxy any longer. It may be related to a new configuration required in nginx (but after a long search we couldn't find any information on the internet) or to another incompatibility.

Cheers,
Pablo

Expected Behavior

Using nginx as reverse proxy to a rosbridge websocket should work normally, as it was working before introducing Autobahn.

Actual Behavior

Connecting to the socket through the nginx proxy returns always HTTP error 400. The connection is not working.

Steps to Reproduce the Problem

To test this you need to start a nginx proxy to manage rosbridge communication.

1. Run ROS in a separate machine (ip) or in a docker container, e.g. ros:kinetic-ros-base-xenial
2. Checkout any master / develop commit after Autobahn was introduced in Autobahn WebSocket server #426, e.g. 1b8396f
3. run the webserver in the port 9091
4. If you connect directly to the ip:9091 then you will see a blue screen with Autobahn

AutobahnPython 0.10.3
I am not Web server, but a WebSocket Endpoint.
You can talk to me using the WebSocket protocol.

For more information, please see:
AutobahnPython

before Autobahn the message was Can "Upgrade" only to "WebSocket".

5. Run nginx docker image as separate container, e.g. as the following docker-compose.yaml:

version: '3'
services:
  nginx:
    image: nginx
    container_name: "nginx"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf
    ports:
      - 80:9091
    logging:
      driver: "json-file"
      options:
        max-file: "5"
        max-size: "10m"
    restart: always

with the following (or similar) nginx.conf (replacing the correct docker-container name or ip in the proxy_pass):

user  nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    sendfile        on;
    keepalive_timeout  65;

    server {

        listen 9091;

        location /test {

            internal;
            resolver 127.0.0.11 ipv6=off; # In order to work in docker, the resolver needs to be this ip to work

            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-NginX-Proxy true;
            proxy_set_header X-Forwarded-Proto $scheme;

            proxy_pass http://ros-docker-name:9091/;
	    proxy_redirect off;
            proxy_read_timeout 86400;

            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade"; 

            proxy_buffers 8 32k;
            proxy_buffer_size 64k;

            reset_timedout_connection on;

        }

    }

}

6. Accessing to the nginx container (port 80) nginx/test will return HTTP error 400. Before Autobahn was working normally. Something is not compatible any longer. It may be related to https://autobahn.readthedocs.io/en/latest/websocket/programming.html?highlight=proxies#server-only-options trustXForwardedFor option. We tried to change it but we couldn't find it in the used version of Autobahn, which we believe to be older. It could also be that Autobahn is not the responsible for this error, but it most likely. Any help will be appreciated!

Specifications

ROS running in a kinetic ros container

• ROS Version (echo $ROS_DISTRO): kinetic
• OS Version (grep DISTRIB_CODENAME /etc/lsb-release): xenial
• Rosbridge Version (roscat rosbridge_server package.xml | grep '<version>'): 0.11.3
• Twisted Version (python -c 'import twisted; print twisted.version'): 16.0.0

[epaezrubio]

Seems like the same issue as in #508. Have you tried setting your websocket_external_port param to 80?

[pablo-quilez]

Hi @epaezrubio, thanks for answering. If I understood correctly you mean to keep port 80 tunneled to 9091 in the docker and add this websocket_external_port param to 80 (as rosparam?). We need to tunnel the connection through a reverse proxy because we add a https layer and a hash validation in URL. I will try to test it in the next days. Our temporally solution was to fix the latest working version because we needed for production.

[rubenanapu]

I had the same problem and solved it with the websocket_external_port arg.

The port 40000 in the host was mapped to 9090 in the container. Below is the full command I used:

roslaunch rosbridge_server rosbridge_websocket.launch _port:=9090 websocket_external_port:=40000 --screen

[caiusno1]

Hey first of all, I encounter the same problem on ros: melodic. I tried to use that arg in my launchfile:

<include file="$(find rosbridge_server)/launch/rosbridge_websocket.launch">
		<param name="port" value="9090"/>
                <param name="websocket_external_port" value="80"/>
	</include>"

traeffic (another reverse proxy) also tell me HTTP error 400. When I look into my terminal where rosbridge run (inside a container) i recognized the following error message:

2021-01-13 18:04:09+0000 [-] failing We│······································
[INFO] [1610560783.317333]: Loading cont│bSocket opening handshake ('missing por│······································
roller: joint_state_controller          │t in HTTP Host header 'localhost' and s│······································
[INFO] [1610560783.347913]: Loading cont│erver runs on non-standard port 9090 (w│······································
roller: joint_trajectory_controller     │ss = False)')                          │······································
[INFO] [1610560783.549021]: Controller S│2021-01-13 18:04:09+0000 [-] dropping c│······································
pawner: Loaded controllers: joint_state_│onnection to peer tcp4:10.42.0.45:38786│······································
controller, joint_trajectory_controller │ with abort=False: missing port in HTTP│······································
[INFO] [1610560783.569391]: Started cont│ Host header 'localhost' and server run│······································
rollers: joint_state_controller, joint_t│s on non-standard port 9090 (wss = Fals│······································
rajectory_controller                    │e)

If you found a solution could you maybe give me a hint :). Thx for your help :)

[mhaboali]

Seems like the same issue as in #508. Have you tried setting your websocket_external_port param to 80?

That helped me a lot, thanks @epaezrubio

[pablo-quilez]

Hello, thank you very much for your answers! Our immediately solution for avoiding production stop was to use an old commit. Recently we updated our system and tested your solution @epaezrubio, it is working fine!