Websocket error after upgrading to 1.3.0

[gsmet]

Hi !

After my upgrade to GitLab 8.3 and Mattermost 1.3.0, I have websocket errors at the top of the page:
Please check connection, Mattermost unreachable. If issue persists, ask administrator to check WebSocket port.

In the Apache server-status, all the requests to the API (GET /api/v1/websocket?session_token_index=0 HTTP/1.1) are stalled (status working).

It was working perfectly before the upgrade so I'm wondering if something was changed?

FYI, I use Apache's mod_proxy_ws_tunnel to upgrade my websocket requests but it used to work perfectly before the upgrade of Mattermost.

Any hints, ideas, advices on how to diagnose the issue?

Thanks!

[coreyhulen]

Nothing has changed with websockets from 1.2 to 1.3. There has been some changes around it, but the underlying connection has stayed the same for several versions. The *.go files haven't been touched in months.

See
https://github.com/mattermost/platform/blob/master/web/react/stores/socket_store.jsx
https://github.com/mattermost/platform/blob/master/api/web_socket.go
https://github.com/mattermost/platform/blob/master/api/web_conn.go

[it33]

Hi @gsmet, please see if this repros if you're on a wired connection instead of Wifi?

I've seen this error intermittently on mobile sometimes when I'm on a bad connection,

[gsmet]

Hi,

We are all using a wired connection (and different ones).

And it's 100% reproducible for all our users on Firefox.

[adrianduke]

I can confirm the same issue, after some hours of uptime (haven't worked out how long yet) the websocket error appears as op stated. I am using jwilder/nginx-proxy with a slightly customised mattermost docker build (just config settings) on v1.3.0. I'm seeing the errors in Chrome 47.0.2526.106 on Mac OS X 10.9. I've just had to restart my mattermost docker container again to remove the warning (second time in 72 hours). Previous builds (v1.1 and v1.2.1) didn't exhibit the same behaviour.

[woutervb]

For me the problem is partially reproducable. In the office environment (ipsec vpn to datacentrum, apache proxy / ssl endpoint) Firefox does have a problem.
But when testing via the internet (with only https) firefox does not exhibit the problem. As I did not upgrade the problem might be caused by another problem.

When using chrome / ie from the same location the problem does not show itself.

[coreyhulen]

Can you look at the browser console and paste any errors you see into this thread. Right Click > Inspect Element > Console

[woutervb]

The lines below appear in the console when clicking on another channel, at which point the blue line, with the websocket error appears

Detected version update refreshing the page bundle-4896.min.js:29:6121
error in updateLastViewedAt status=error statusCode=0 err= bundle-4896.min.js:29:19796
Object { message: "Please check connection, Mattermost…", connErrorCount: 4 } bundle-4896.min.js:29:19813
error in getPosts status=error statusCode=0 err= bundle-4896.min.js:29:19796
Object { message: "Please check connection, Mattermost…", connErrorCount: 5 } bundle-4896.min.js:29:19813
error in getChannelExtraInfo status=error statusCode=0 err= bundle-4896.min.js:29:19796
Object { message: "Please check connection, Mattermost…", connErrorCount: 6 } bundle-4896.min.js:29:19813
error in getPosts status=error statusCode=0 err= bundle-4896.min.js:29:19796
Object { message: "Please check connection, Mattermost…", connErrorCount: 7 } bundle-4896.min.js:29:19813
error in findTeams status=error statusCode=0 err= bundle-4896.min.js:29:19796
Object { message: "Please check connection, Mattermost…", connErrorCount: 8 } bundle-4896.min.js:29:19813
error in getMyTeam status=error statusCode=0 err= bundle-4896.min.js:29:19796
Object { message: "Please check connection, Mattermost…", connErrorCount: 9 } bundle-4896.min.js:29:19813
error in getAllPreferences status=error statusCode=0 err= bundle-4896.min.js:29:19796
Object { message: "Please check connection, Mattermost…", connErrorCount: 10 } bundle-4896.min.js:29:19813
Firefox can't establish a connection to the server at wss://chat.xxx.xxx/api/v1/websocket?session_token_index=0. bundle-4896.min.js:28:18936
downloadable font: download failed (font-family: "Open Sans" style:normal weight:normal stretch:normal src index:3): status=2152398850 source: https://chat.xxx.xxx/static/fonts/open-sans-v13-latin-ext_latin_cyrillic-ext_greek-ext_greek_cyrillic_vietnamese-regular.woff2
downloadable font: download failed (font-family: "Glyphicons Halflings" style:normal weight:normal stretch:normal src index:1): status=2152398850 source: https://chat.xxx.xxx/static/fonts/glyphicons-halflings-regular.woff2 bootstrap-3.3.5.min.css:5:3022
downloadable font: download failed (font-family: "Open Sans" style:normal weight:600 stretch:normal src index:3): status=2152398850 source: https://chat.xxx.xxx/static/fonts/open-sans-v13-latin-ext_latin_cyrillic-ext_greek-ext_greek_cyrillic_vietnamese-600.woff2
downloadable font: download failed (font-family: "Open Sans" style:normal weight:bold stretch:normal src index:3): status=2152398850 source: https://chat.xxx.xxx/static/fonts/open-sans-v13-latin-ext_latin_cyrillic-ext_greek-ext_greek_cyrillic_vietnamese-700.woff2
downloadable font: kern: Too large subtable., table discarded (font-family: "Open Sans" style:normal weight:normal stretch:normal src index:3) source: https://chat.xxx.xxx/static/fonts/open-sans-v13-latin-ext_latin_cyrillic-ext_greek-ext_greek_cyrillic_vietnamese-regular.woff2 google-fonts.css:28:12
downloadable font: kern: Too large subtable., table discarded (font-family: "Open Sans" style:normal weight:600 stretch:normal src index:3) source: https://chat.xxx.xxx/static/fonts/open-sans-v13-latin-ext_latin_cyrillic-ext_greek-ext_greek_cyrillic_vietnamese-600.woff2 google-fonts.css:54:12
Content Security Policy: Interpreting none as a hostname, not a keyword. If you intended this to be a keyword, use 'none' (wrapped in single quotes).
mutating the [[Prototype]] of an object will cause your code to run very slowly; instead create the object with the correct initial [[Prototype]] value using Object.create react-bootstrap-0.28.1.min.js:6:2597
"websocket connecting to wss://chat.xxx.xxx/api/v1/websocket?session_token_index=0" bundle-4896.min.js:28:18893
downloadable font: kern: Too large subtable., table discarded (font-family: "Open Sans" style:normal weight:normal stretch:normal src index:3) source: https://chat.xxx.xxx/static/fonts/open-sans-v13-latin-ext_latin_cyrillic-ext_greek-ext_greek_cyrillic_vietnamese-regular.woff2
downloadable font: kern: Too large subtable., table discarded (font-family: "Open Sans" style:normal weight:600 stretch:normal src index:3) source: https://chat.xxx.xxx/static/fonts/open-sans-v13-latin-ext_latin_cyrillic-ext_greek-ext_greek_cyrillic_vietnamese-600.woff2
downloadable font: kern: Too large subtable., table discarded (font-family: "Open Sans" style:normal weight:bold stretch:normal src index:3) source: https://chat.xxx.xxx/static/fonts/open-sans-v13-latin-ext_latin_cyrillic-ext_greek-ext_greek_cyrillic_vietnamese-700.woff2 google-fonts.css:80:12

[woutervb]

An additional update, when attempting to dig a little further in de console I found the following lines, which might be relevant.

error in getProfiles status=error statusCode=0 err= bundle-4896.min.js:29:19796
Object { message: "Please check connection, Mattermost…", connErrorCount: 12 } bundle-4896.min.js:29:19813
Firefox can't establish a connection to the server at wss://chat.xxx.xxx/api/v1/websocket?session_token_index=0. bundle-4896.min.js:28:18936

[gsmet]

As for me, the upgrade of the connection to websocket goes OK but then I have the following in the console:
The connection to wss://team.openwide.fr/api/v1/websocket?session_token_index=0 was interrupted while the page was loading.

Nothing in the Mattermost log.

[coreyhulen]

@woutervb looks like the proxy isn't configured correctly. Most of the errors are referring to std http get/post calls and have nothing todo with the websocket. For example getProfiles, updateLastViewedAt, getPosts, etc have nothing todo with the websocket. Also seeing Detected version update refreshing every time means the X-Version-Id header property isn't getting passed through your proxy.

[coreyhulen]

@gsmet I think your issue is different. The error message you shared is coming from firefox and not mattermost. I'm thinking it might be this issue https://bugzilla.mozilla.org/show_bug.cgi?id=712329. What version of firefox are you using? what OS? any non-std stuff installed like adblocker?

[coreyhulen]

@woutervb you're hitting https://github.com/mattermost/platform/blob/master/web/react/utils/async_client.jsx#L55 which is trying to call window.location.reload(true); for every click on a channel name since it thinks a new version was installed, but what's weird is the connection fails for all std http calls. When clicking on a new channel does the browser address bar change? maybe https is getting stripped and re-written as http?

[woutervb]

@coreyhulen I'm not sure what you exacly mean. The headers I can check at a later moment that remark is clear. My setup is a proxy that only handles https, and does not even respond to http requests, using the apache recipie found at https://mattermost.atlassian.net/browse/PLT-635, only using a different ip adres for the mattermost installation.
And using ProxyPassReverse ws://192.168.1.85:8065/api/v_1_/websocket and not ProxyPassReverse ws://127.0.0.1:8065/api/v_i_/websocket.

When clicking a channel the url bar of the browser changes to the new channel name, but I am not sure that, that is what you are referring to.

[gsmet]

@coreyhulen I did find this Firefox bug but what bugs me is that I have the problem even if launch a brand new instance of Firefox. Moreover, it's definitely related to the Mattermost upgrade to 1.3 as we didn't have the problem before and nothing else has changed on our setup.

That being said, to be on the safe side, maybe you could include the workaround described here to Mattermost: https://bugzilla.mozilla.org/show_bug.cgi?id=712329#c12 .

All the users at my company have this issue.

My install: Firefox 42, Ubuntu Trusty, Adblock edge but I tried to disable it and the problem is still there.

[woutervb]

@coreyhulen Just checked the headers, and see that X-version-Id is received on the requests (and a pragma: no-cache on the static elements, which striked me odd, but is unrelated)

[coreyhulen]

@woutervb would you mind sharing some of the network traffic as seen by firefox? Please feel free to blur/remove any sensitive data and/or direct message me at pre-release.mattermost.com corey or via email at corey at mattermost dot com. If you can show the initial load and then click clear then show switching a channel. Also showing the X-Version-ID response value. A screen shot is probably good enough or you can right click the traffic and save all as har and send me that. Note the har file will have a bunch of sensitive data.

[coreyhulen]

@woutervb something like

[coreyhulen]

@gsmet yeah, we'll start on the workaround. I'm also not sure what has changed between 1.2 and 1.3 that would cause the issue. I've been looking at the history of https://github.com/mattermost/platform/blob/master/web/react/stores/socket_store.jsx and cannot see any substantive changes that would be causing the issue.

Can you connect a firefox browser directly too the box bypassing the proxy as a test? I just want to rule out an any apache proxy issue.

I think @woutervb problem is different. I'm still leaning towards an apache config issue with their vpn environment for him because of the comment when testing via the internet (with only https) firefox does not exhibit the problem

[coreyhulen]

@gsmet the only change I see is how we clear the error 75a2ce0. Maybe it's reconnecting and the error for the blue bar isn't getting cleared. After you see the connection to wss://team.openwide.fr/api/v1/websocket?session_token_index=0 was interrupted while the page was loading. do you have any another message like websocket connecting to wss://team.openwide.fr/api/v1/websocket?session_token_index=0 or websocket re-established connection further down in the browser console log?

[adrianduke]

One of our users was getting websocket connecting to wss://xxx.xxxxxx.io/api/v1/websocket?session_token_index=0 in their console whilst the blue bar appeared. They were chatting whilst it (blue bar) was present in their browser as well.

[gsmet]

@coreyhulen

The websocket error also occurs when I'm not passing through a proxy.

I don't have any other message regarding websockets in the console.

Here are the queries executed on a regular basis (I don't know if it states that websockets actually are working or not):

POST XHR https://team.openwide.fr/api/v1/users/status [HTTP/1.1 200 OK 13ms]
GET XHR https://team.openwide.fr/api/v1/channels/upuzmj9bd3ynzn58jwf6nheg9o/ [HTTP/1.1 200 OK 14ms]
POST XHR https://team.openwide.fr/api/v1/users/status [HTTP/1.1 200 OK 27ms]
POST XHR https://team.openwide.fr/api/v1/users/status [HTTP/1.1 200 OK 15ms]
POST XHR https://team.openwide.fr/api/v1/users/status [HTTP/1.1 200 OK 16ms]

[woutervb]

Sorry for the long delay. I missed that there where updates.
Tried to reproduce today on a fresh Fedora 23 installation with default firefox and could not reproduce the above mentioned symptoms.

I did change my apache proxy configuration by adding:

RequestHeader unset If-Modified-Since
RequestHeader unset If-None-Match

Which removed the symptoms of double loading on channel switch. So as far as I can see I have a fully functional setup, with apache reverse proxy, handling name based ssl.
If there is interest I could document the configuration (and hopefully help others that way).
Please let me know.

P.S. Found the above mentioned change on: http://forum.mattermost.org/t/too-many-detected-version-update-refreshing-the-page-when-proxying-with-apache/688

[coreyhulen]

@woutervb we would love some documentation for what worked with apache.

[coreyhulen]

@gsmet I've implemented the firefox workaround with #2094 and it should appear in our next release.

[gsmet]

@coreyhulen Nice, thanks!

[it33]

@gsmet, @woutervb @adrianduke thanks to all for reporting issues and helping advance the conversation.

It seems:

1. There are different configurations triggering the same error message.
2. There is a fix that should be available in Mattermost 2.0 (thanks @coreyhulen!)
3. There is now documentation to help reduce the chance of this issue in future (thanks @woutervb!)

I'm closing this ticket and would ask the help of people on the thread to see if the mix of Mattermost 2.0 and the new configuration instructions solves the issue. If not, please open a new ticket with repro steps on your specific configuration.