Unable to connect over webockets with TLS (wss) from Firefox

[stepanmracek]

Hi,
I'm running Mosquitto v1.5.8 with enabled websockets, but I'm unable to connect from Firefox when the broker is running over TLS. Strange thing is that It works from Chrome/Chromium. I tried both MQTT.js as well as Paho javascript client libraries with no success.

Connecting always ends with error Firefox can’t establish a connection to the server at wss://localhost:9001/mqtt.

Environment:

• Mosquitto: 1.5.8
• libwebsockets: 3.1.0
• openssl: 1.1.1.b

Here is my broker configuration:

allow_anonymous false
password_file /home/stepo/tmp/mqtt_pwd

listener 9001
protocol websockets

cafile /home/stepo/tmp/mqtt-certs/ca.crt
certfile /home/stepo/tmp/mqtt-certs/server.crt
keyfile /home/stepo/tmp/mqtt-certs/server.key

(When I comment the cafile, certfile and keyfile entries and thus running on ws:/ everything in Firefox works)

Here you can find GIST for JS Paho client: https://gist.github.com/stepanmracek/9b6d4a6ee9ef9d4d6efe738c069a9577
I always outputs following error to the console: AMQJS0007E Socket error:undefined.

HINT

• I've also tried Mosquitto 1.4.11 with libwebsockets 1.6.3 and it works
• On another machine I have Mosquitto 1.4.15 but this time with libwebsockets 3.0.0 and there I'm facing the same issue -> does not work from Firefox.

Therefore I'm suspecting that this is some regression in libwebsockets that is unfortunately reproducible only in Firefox.

[ralight]

Thanks for the hints, I'll take a look. I'd suggest using libwebsockets 2.4.x, that has proved to be a pretty reliable version.

[timothy-godfrey]

Hi!
I can confirm similar behaviour: secure Websockets works between chrome/chromium and Mosquitto, but not between Firefox(v68) and Mosquitto, using Mosquitto v1.6.4 and libwebsockets-3.1. For the time being I will revert to libwebsockets-2.4

I have uploaded a couple of filtered Wireshark captures to Firefox Send here one for Firefox not working and one for Chrome working. Chrome appears to be using Websockets over TLS, whereas Firefox appears to be using Websockets over http2.

In the Firefox capture things seem to go awry at packet 34 when the Mosquitto host (192.168.3.28) sends RST_STREAM followed by GO_AWAY. The host client running firefox (192.168.3.23) replies with its own GO_AWAY (for the control stream?), and after that a TLS 'Close Notify' Alert. Then the Mosquitto host sends the TCP RST packets.

I don't know how to get more debugging information without rebuilding Mosquitto and libwebsockets, so I don't have any insights into what is causing the Mosquitto host to reset the http2 stream. My impression is that something from Firefox is malformed/unexpected from the libwebsockets point of view and so libwebsockets responds by resetting the stream.

I hope that is useful!

[timothy-godfrey]

Writing down the above led me to consider that it was a WSS over http/2 problem. So I recompiled libwebsockets-3.1 with http2 support disabled, and now it works! I can connect to my Mosquitto broker over WSS from Firefox and Chrome.

[ralight]

@timothy-godfrey Are you able to try the current mosquitto fixes branch with libwebsockets compiled with http2 support? There are some libwebsockets related changes in fixes. Or maybe with libwebsockets 3.2?

[vincent-lefevere]

Today, I download the last version of libwebsockets-3.2 and I compile it with http2 support. Next, I compile mosquitto (version 1.6.8). Result: Firefox(V71.0) does not want to use mqtt over wss. No trouble with Chrome!
When I compile libwebsockets-3.2 (as well as libwebsocket-3-1) without http2 support and next I compile mosquitto, then Firefox work as well as Chrome.

[jonjonarnearne]

You can workaround this in firefox by setting network.http.spdy.websockets to false in about:config.

This seems to be a bug in mosquitto, as the libwebsockets-test-server is working fine with wss over http/2.

[jonjonarnearne]

Just tested this with a connection to wss://test.mosquitto.org:8081/mqtt. And this fails if we are using wss over http/2.

[jonjonarnearne]

The problem seems to be be that libwebsockets claim to support http/2, so firefox tries that,
but then mosquitto closes the connection?

[ralight]

@jonjonarnearne libwebsockets on test.mosquitto.org isn't compiled with http2 support, so if it's advertising it as available, that was a bug in libwebsockets. I use firefox all the time, and don't have a problem connecting to https://test.mosquitto.org/sys/ssl.html , for example - I haven't modified about:config.

What is the best way for me to test this?

One thing that may be causing an issue with lws, is that in more recent versions the WITH_EXTERNAL_POLL option is turned off by default, so Mosquitto does not receive any callbacks for the network traffic.

[jonjonarnearne]

@ralight It seems you are completely right.
I'm not able reproduce the failing connection to wss://test.mosquitto.org:8081/mqtt.

There must have been something wrong with my test setup when this connection failed on me.

I'm sorry for the noise.

[gary-hammer]

This is an issue on my website where I use MQTT(wss) to show live data.
With Firefox, unless I use the about:config workaround, the connection is reset and the data is not updated.
If I use the about:config workaround, the connection is established and data is updated.

The problem is, users of the website aren't going to dig to find out why, they'll just leave.

[wiebeytec]

I wiresharked some TLS websocket traffic to a server of ours that has this issue. Ubuntu 20.04, libwebsocket 3.2.1-3, mosquitto 1.6.12, compiled ourselves, using standard Ubuntu libs.

Chromium never tries http2. Firefox does (by starting with PRI * HTTP/2.0\r\n\r\nSM). There are also answers in http2, so it's not just a bug in reporting it can handle http2 while it can't.

Could be the WITH_EXTERNAL_POLL. I don't know how Ubuntu's version is compiled, nor do I have an MQTT packet parser handy to see where it gets stuck.

I fixed it by putting an Nginx proxy in front of it, proxying using HTTP 1.1. I can afford it on this server, but it's not ideal, because you'll run into maximum number of connections, because of TCP ephemeral port exhaustion.

[ralight]

@wiebeytec Is there anything special to reproduce it? The easiest you can make it for me so I don't have to guess anything would be great.

The Ubuntu package doesn't have external poll support. I know there are problems around the 3.2.x versions, there are version specific fixes in Mosquitto to deal with them, but I know people have still had other problems with them.

[wiebeytec]

No, just use Ubuntu 20.04, install buildessential and libsockets-dev, compile 1.6.12 and use Firefox and look in the console (CTRL-SHIFT-K). I didn't save the error messages unfortunately.

There may be a chance it only happens after a few messages. I don't have any other basis to say this than that our webapp got past the first step and then stalled. If you can't reproduce it, you can contact me privately, and I may be able to set up a dev server and have you connect to it.

[mister-teddy]

Hello everyone, what is the status of this issue? I'm having a project using mqtt over https. And our Firefox user have to use the about:config workaround in order to use the website 😞