Websocket connection not possible through apache2 when updating from 0.11.3 to 0.11.4

[victoririzarx3d]

Our setup: Ros melodic 1.14.5 , running in ubuntu bionic 18.04.1 LTS, apache2 2.4.29, rosbridge_server 0.11.3 (working), rosbridge_server 0.11.4-0.11.5 (not working), python 2.7.17

We currently implement rosbridge_server to serve a couple of websockets that we use to connect the front end of our application to the ROS backend. We use apache as a reverse proxy to route the requests from an specific outer endpoint to our locally served websockets (e.g. ProxyPass /api ws://localhost:9090).

An example of our apache2 setup is seen below

<VirtualHost *:80>
  ProxyPreserveHost On
  ProxyRequests Off
  <Location /api>
    ProxyPass ws://localhost:9090 retry=0 keepalive=On
  </Location>
</VirtualHost>

We realized that upon updating rosbridge_server to 0.11.4, which as I understand changes the websocket backend in python to use the Autobahn library, our apache proxying stopped working and we could not get access to the web socket from the outer world. The interesting thing is that the websocket itself works correctly and we can have access to it if we connect directly, bypassing apache (e.g can connect to ws://<IP_OF_MACHINE>:9090).

Our launch scripts for the rosbridge websocket can be seen below

<launch>

  <include ns="customer" file="$(find rosbridge_server)/launch/rosbridge_websocket.launch">
    <!-- use default port -->
    <arg name="port" value="9090" />
    <arg name="topics_glob" default="[*]" />
    <arg name="services_glob" default="[*]" />
    <arg name="params_glob" default="[]" />
  </include>

</launch>

We have been throwing arrows in the dark for a while trying to see what has changed. One thing we have noticed is that with rosbridge_server 0.11.3, launching a rosbridge_websocket node binds a python process to IPV4 and IPV6 addresses, as seen by checking with the command below

user@webvenice: : sudo ss -lptn 'sport = :9090'

State             Recv-Q             Send-Q                           Local Address:Port                           Peer Address:Port                                                             
LISTEN            0                  128                                    0.0.0.0:9090                                0.0.0.0:*                 users:(("python",pid=24777,fd=14))             
LISTEN            0                  128                                       [::]:9090                                   [::]:*                 users:(("python",pid=24777,fd=15))  

In contrast, when testing with 0.11.4 or 0.11.5 we see that only that there is a single python process binding IPV4 addresses

user@webvenice: sudo ss -lptn 'sport = :9090'

 State             Recv-Q        Send-Q                           Local Address:Port                  
Peer Address:Port                      LISTEN            0                  50                        
0.0.0.0:9090                                0.0.0.0:*                 users:(("python",pid=24777,fd=14))

We are sure that apache2 works and routes correctly the outer request but when routing to the local network it has trouble, as seen in the logs below

[Mon Apr 20 14:11:38.952259 2020] [proxy:error] [pid 601:tid 140657578448640] (111)Connection refused: AH00957: WS: attempt to connect to 127.0.0.1:9090 (localhost) failed [Mon Apr 20 14:11:38.952292 2020] [proxy_wstunnel:error] [pid 601:tid 140657578448640] [client ::1:62928] AH02452: failed to make connection to backend: localhost [Mon Apr 20 14:11:39.960145 2020] [proxy:error] [pid 601:tid 140657477801728] (111)Connection refused: AH00957: WS: attempt to connect to 127.0.0.1:9090 (localhost) failed [Mon Apr 20 14:11:39.960176 2020] [proxy_wstunnel:error] [pid 601:tid 140657477801728] [client ::1:62974] AH02452: failed to make connection to backend: localhost [Mon Apr 20 14:11:40.961435 2020] [proxy:error] [pid 602:tid 140656999646976] (111)Connection refused: AH00957: WS: attempt to connect to 127.0.0.1:9090 (localhost) failed [Mon Apr 20 14:11:40.961465 2020] [proxy_wstunnel:error] [pid 602:tid 140656999646976] [client ::1:63024] AH02452: failed to make connection to backend: localhost [Mon Apr 20 14:11:41.963662 2020] [proxy:error] [pid 601:tid 140657469409024] (111)Connection refused: AH00957: WS: attempt to connect to 127.0.0.1:9090 (localhost) failed [Mon Apr 20 14:11:41.963686 2020] [proxy_wstunnel:error] [pid 601:tid 140657469409024] [client ::1:63062] AH02452: failed to make connection to backend: localhost [Mon Apr 20 14:11:42.964781 2020] [proxy:error] [pid 601:tid 140658559936256] (111)Connection refused: AH00957: WS: attempt to connect to 127.0.0.1:9090 (localhost) failed [Mon Apr 20 14:11:42.964809 2020] [proxy_wstunnel:error] [pid 601:tid 140658559936256] [client ::1:63108] AH02452: failed to make connection to backend: localhost [Mon Apr 20 14:11:43.970101 2020] [proxy:error] [pid 602:tid 140658408933120] (111)Connection refused: AH00957: WS: attempt to connect to 127.0.0.1:9090 (localhost) failed [Mon Apr 20 14:11:43.970128 2020] [proxy_wstunnel:error] [pid 602:tid 140658408933120] [client ::1:63128] AH02452: failed to make connection to backend: localhost [Mon Apr 20 14:11:44.978513 2020] [proxy:error] [pid 601:tid 140658543150848] (111)Connection refused: AH00957: WS: attempt to connect to 127.0.0.1:9090 (localhost) failed [Mon Apr 20 14:11:44.978543 2020] [proxy_wstunnel:error] [pid 601:tid 140658543150848] [client ::1:63168] AH02452: failed to make connection to backend: localhost [Mon Apr 20 14:11:45.988754 2020] [proxy:error] [pid 602:tid 140658559936256] (111)Connection refused: AH00957: WS: attempt to connect to 127.0.0.1:9090 (localhost) failed [Mon Apr 20 14:11:45.988777 2020][proxy_wstunnel:error] [pid 602:tid140658559936256] [client ::1:63258] AH 02452: failed to make connection to backend: localhost

We have tried to tweak our apache2 setup with different configurations (Proxy allow all, ProxyPassReverse, instead of binding the ws socket to localhost to 0.0.0.0 or 127.0.1, etc) and also tried to tweak the launch script of the websockets (changing the address explicitly to 0.0.0.0 or 127.0.0.1, changing ports,etc) but nothing works with rosbridge_server 0.11.4 or 0.11.5

Do you have any experience with a similar setup that could shed some light? Is there anything we are missing in how should we setup the websocket with 0.11.4 and above?

Thanks in advance

[gavanderhoorn]

Posted first on ROS Answers (Issues with apache2 and updating rosbridge_server from 0.11.3 to 0.11.4), but this seems like it would belong here more.

[mvollrath]

You might try the websocket_external_port arg, setting it to the external port (which looks like :80) may help.

[moriarty]

Likely due to #426

[victoririzarx3d]

Hi guys, thanks for the fast response.

I forgot to mention that I tried @mvollrath suggestion setting the websocket_external_port to 80 and that did not work also. But #426 has a pretty interesting description of the breaking changes and could be that our roslibjs frontend is not sending the Origin header with the schema information, making the websocket connection unusable. I will make a test setup in our system and give this a try and let you know the results

[github-actions]

This issue has been marked as stale because it has been open for 180 days with no activity. Please remove the stale label or add a comment to keep it open.