Failed to upgrade websocket connection #5386
Comments
|
Thanks @demedos for the report, Did you try these troubleshooting steps? https://docs.mattermost.com/install/troubleshooting.html?highlight=unreachable#please-check-connection-mattermost-unreachable-if-issue-persists-ask-administrator-to-check-websocket-port Also, could we have your help posting this to the Troubleshooting Forum? There are more forum visitors than GitHub issue visitors, which means a potentially faster response. Please link back to your new post from this issue? |
|
It seems that the problem is about IIS 7.5 not supporting WebSockets, as stated here:
Unluckly the only solutionn is to upgrade, so tomorrow I'll try out a reverse proxy with a newer version and will let you know if it's solved |
|
I would like to close the issue but I've had no luck so far. for testing purposes I've added only 1 rewrite rule to IIS, defined as below
Now that means that whatever is passed before "://" will be used as protocol in the request to the internal machine, so http remains http and ws remains ws, so the proxy works fine. The logs are repeatedly reporting
and the Developer Console But in the System Console from Mattermost CORS is enabled. What could possibly be wrong? This is quite a show-stopper for me |
|
Could this be related to the fact that the log shows |
|
This is a problem with a recent change to web socket processing in Mattermost. Specifically, 1a9891f Try reverting the change to CheckOrigin. This really should mimic what is being done for HTTP connections in the server.go file (see ServeHTTP where there is the check for Origin header). We had the same problem recently and reverting this change fixed our web socket issues. |
|
I'm a windows user, any chance you branch and compile it? |
|
@demedos @BRH-Enrian are you using multiple Mattermost URLs through proxy forwarding? If so, what is your We have a ticket to investigate a cross-origin issue introduced by 3.6.2: https://mattermost.atlassian.net/browse/PLT-5635 |
|
@jasonblais What do you mean by multiple Mattermost URLs? My current flow is My It seems like this has something to do with the fact that Application Request Routing is stripping off the I hope this will be fixed asap as this is a showstopper for us since IIS is the only available web server. |
|
IMO, the issue is that the default behavior for web socket processing by Mattermost does not honor CORS setting in the Mattermost config. The code used to accept *any* connection request, but now it relies on default behavior as documented at http://www.gorillatoolkit.org/pkg/websocket <http://www.gorillatoolkit.org/pkg/websocket> under “Origin Considerations” which states that
If the CheckOrigin field is nil, then the Upgrader uses a safe default: fail the handshake if the Origin request header is present and not equal to the Host request header.
This is clearly not going to work if you have a cross-origin situation. The best solution would be to adopt similar processing as is done in api/context.go:
if len(*utils.Cfg.ServiceSettings.AllowCorsFrom) > 0 {
origin := r.Header.Get("Origin")
if *utils.Cfg.ServiceSettings.AllowCorsFrom == "*" || strings.Contains(*utils.Cfg.ServiceSettings.AllowCorsFrom, origin) {
w.Header().Set("Access-Control-Allow-Origin", origin)
if r.Method == "OPTIONS" {
w.Header().Set(
"Access-Control-Allow-Methods",
strings.Join(allowedMethods, ", "))
w.Header().Set(
"Access-Control-Allow-Headers",
r.Header.Get("Access-Control-Request-Headers"))
}
}
}
where instead of setting headers just return true.
Brad
… On Mar 6, 2017, at 10:24, Alexander ***@***.***> wrote:
@jasonblais <https://github.com/jasonblais> What do you mean by multiple Mattermost URLs? My current flow is
Externail IP -> IIS Server -> Linux Server.
My SiteURL property was set as https://website.com.
About the CORS, it's not working with the * nor with https://website.com.
It seems like this has something to do with the fact that Application Request Routing is stripping off the Request-Origin header from the request, or something like that.
http://stackoverflow.com/questions/34316825/websockets-reverse-proxy-in-iis-8 <x-msg://1/url>
I hope this will be fixed asap as this is a showstopper for us since IIS is the only available web server.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#5386 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AX8Bpr4vPL57Vbt_Omh0TcxFGng79iKhks5ri9DrgaJpZM4L_eH6>.
|
|
@BRH-Enrian That makes sense now, it almost drove me crazy. That means that it doesn't matter the Web server, the cross-origin requests will be refused. |
|
See if this works for you: https://github.com/bradhowes/platform/releases/tag/v3.6.1-websocket-cors |
|
@bradhowes Thank you, but as soon I click the |
|
The build contains a stock config.json file which will overwrite anything you may already have there if you just blast it into /opt/mattermost. Sorry. Best bet is to untar in a tmp directory and then move the bin/platform executable into /opt/mattermost/bin. Alternatively, save the config.json file first, do what you did, and then restore the file. The build was packaged using Mattermost own Makefile. |
|
That's exactly what I did. Saved the config.json, replaced all and then restored it. Anyway, changing the value manually works, the value is shown in the system console. Now the error is changed:
This is the web request sent from the browser This may be related to the stack overflow post i mentioned before |
|
@bradhowes Any way to tell it this is the correct header?
|
|
That looks OK to me. This is what is giving you an error? I’m not an expert on WebSocket protocol, but that appears normal. Note that from https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API/Writing_WebSocket_servers <https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API/Writing_WebSocket_servers> the Sec-WebSocket-Accept value should have been calculated from the Sec-WebSocket-Key value:
The Sec-WebSocket-Accept part is interesting. The server must derive it from the Sec-WebSocket-Key that the client sent. To get it, concatenate the client's Sec-WebSocket-Key and "258EAFA5-E914-47DA-95CA-C5AB0DC85B11" together (it's a "magic string <https://en.wikipedia.org/wiki/Magic_string>"), take the SHA-1 hash <https://en.wikipedia.org/wiki/SHA-1> of the result, and return the base64 <https://en.wikipedia.org/wiki/Base64> encoding of the hash.
You could follow the above to verify that the received 'accept' value matches the given 'key' value. If not, then something is wrong in your connection flow.
Brad
… On Mar 7, 2017, at 12:16, Alexander ***@***.***> wrote:
@bradhowes <https://github.com/bradhowes> Any way to tell it this is the correct header?
<https://cloud.githubusercontent.com/assets/16702156/23654215/eac8aec8-032f-11e7-8bee-1cc534c08830.png>
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#5386 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAp7YlRx13JmiEOx6uAQpJbF58g4pFBQks5rjTyrgaJpZM4L_eH6>.
|
|
You are right, it doesn't match. Given
While my browser gets this Response Header: I should test the connection from whitin the reverse proxy server I suppose. |
|
@bradhowes |
|
@bradhowes would your PR resolve the issues described here? #5667 |
|
@jasonblais Not sure. @demedos reported that something worked under Nginx, but not clear if that was with my patch or not. |
|
@jasonblais The patch @bradhowes provided here worked for me, but I have to edit the config file manually because the web system console is not persisting my changes after I hit the save button, not sure if it is a normal behaviour using that patch. Moreover, it is working using Nginx, because (at least) IIS 8.5 in Windows Server 2012, more specifically its Application Request Routing (ARR) module used to do reverse proxy, has a bug which tells the browser that he supports the per-message compression So yes, the patch is working, but IIS 8.5 is not supported because of its module bug (ref. here). |
|
I am seeing the exact same error/behavior with my installation. I am using Apache for the reverse proxy. One other difference is I have TLS enabled so my error is with wss:// I tried the patch provided above but the issue did not go away. Please, let me know if I should file a new issue, where I can attach configuration and screen-captures. |
|
@attzonko Have you edited the config file manually? Do you see the correct value in the System Console after editing the |
|
@demedos I assume you mean when I tried the patch above. Yes I did edit the Since my post I ended up installing Nginx alongside Apache, and have nginx doing the reverse proxy with SSL and the upgrade for websocket and it is indeed working. Nginx passes through all my legacy apps traffic on to Apache. |
|
@attzonko Yes, I meant that. What was the exact error you had though? I'd still like to know if ARR is being buggy. |
|
That was exactly the error I had. Here is a quick cut/paste from my logs: |
|
Here is what my apache config looked like: |
|
I've upgraded to 3.7.1 and nothing broke, even if the websocket.go has the
|
Follow-up from - https://mattermost.atlassian.net/browse/PLT-5635 - mattermost/mattermost#5667 - mattermost/mattermost#5386 Also fix two broken links
|
Hi all, PR to add websocket CORS support got merged last Thursday: #5667 This should help resolve most of the issues described here. We've also created a PR to add the following troubleshooting steps to our install guide: Please let me know if there's anything else to address, otherwise we'll close the issue in a couple of days. |
|
In IIS WebSockets must be enabled, since the option is not on by default. Maybe it could help in the documentation. Right now I'm using NGINX so afaik that's all |
|
Ah okay, I'll add that note as well, doesn't hurt. Thanks @demedos! |
|
Erm. Read it all... since I got the same error. Edit: Running Community 3.7.3 on IIS 8 with MySql (Websockets enabled and the Rewrite Rules from Unofficial Enhancements - Production Install on Windows 2012) Edit 2: Ah, I see. A bug in IIS 8.x with ARR |
|
In regard to:
Can it be fixed to this in a close future version? |
|
AWESOME. Works for me as of 3.8.2 |
|
Sorry to bring this thread up again. I am using mattermost 4.9.0 with apache 2.4. Trying to setup https proxy via apache and no luck. I followed and tried all of the solutions above. I supposed all those patches are included in 4.9.0 already. Anyone would kindly provide the "ServiceSettings" part of the config.json for this setup please? My goal is: access mm via https://hostname.domain.com:443 and apache proxy all traffic to http://127.0.0.1:8065 and ws://127.0.0.1:8065. Thank you. |
|
Hi @tstkenny, Thanks for your feedback, As this thread is very old and was closed, may I ask you to open a new issue, stating what you have tried, which set-up documentation you have followed and Mattermost server version? Perhaps you can also include the logs which will help us to troubleshoot. This way, it will be easier for our devs to help. Also, we don't officially support Apache but here are the unofficial docs we have - not sure whether you have looked at them?
For continuity, please link in this issue to the new issue you open. |
|
In a nutshell, make sure you have: IIS 8 + ARR 3 + WebSockets IIS extension. Then here's a sample <?xml version="1.0" encoding="UTF-8"?>
<configuration>
<system.webServer>
<rewrite>
<rules>
<rule name="Mattermost Inbound" stopProcessing="true">
<match url="(.*)" />
<action type="Rewrite" url="{C:1}://localhost:8065/mattermost/{R:1}" />
<conditions logicalGrouping="MatchAll" trackAllCaptures="true">
<add input="{CACHE_URL}" pattern="^(.+)://" />
<add input="{HTTP_HOST}" pattern="^somedomain\.com$" />
</conditions>
<serverVariables>
<set name="HTTP_SEC_WEBSOCKET_EXTENSIONS" value="client_max_window_bits" />
</serverVariables>
</rule>
</rules>
</rewrite>
<security>
<requestFiltering allowDoubleEscaping="true" />
</security>
</system.webServer>
</configuration> |
Summary
After setting up a reverse proxy the log is full of those messages:
[2017/02/13 08:53:13 PST] [EROR] websocket connect err: websocket: could not find connection header with token 'upgrade'
[2017/02/13 08:53:13 PST] [EROR] /api/v3/users/websocket:connect code=500 rid=czz3h9kqtjfb9jz3jo1sqmujne uid=1erjogmb7jraucpy1e1wifcmfy ip=192.168.10.1:56489 Failed to upgrade websocket connection [details: ]
Steps to reproduce
Set up mattermost and do a reverse proxy with IIS
Expected behavior
No error messages
Observed behavior
My website, abc.company.com goes on a Windows Server with IIS, passes the request to an internal ubuntu server machine 192.168.x.x:8065. Everything is fine except those ws:// error messages
The error messages:
The reverse proxy on IIS:
The error message appearing on the website:
Possible fixes
--
The text was updated successfully, but these errors were encountered: