dart:io client WebSocket can not connect to some WebSocket servers

[vadimtsushko]

Some problem description is at https://groups.google.com/a/dartlang.org/forum/#!topic/misc/hPBiK2OlLs0

Specifically I have a trouble connection to Qlik Sense Engine API WebSocket service.
I believe problem is Qlik Sense expects handshake header keys in capitalized form and
dart:io HttpHeaders keys are always converted to lowercase.

[whesse]

@sgjesse @mkustermann
Here is the relevant information from the email thread: The problem seems to be that dart:io lowercases header names, and Qlik web socket code only accepts mixed case header names:

Vadim said:
So I spent some time with WireShark to find what is different in WebSocket handshaking of nodejs (successful) and dart:io (unsuccessful) communication. For now I'm almost totally sure that difference and reason why dart:io WebSocket connections break with error is that dart:io header keys are converts to lowercase.

Successful communication looks like:

Here is the relevant information from the email thread: The problem seems to be that dart:io lowercases header names, and Qlik web socket code only accepts mixed case header names:

Vadim said:
So I spent some time with WireShark to find what is different in WebSocket handshaking of nodejs (successful) and dart:io (unsuccessful) communication. For now I'm almost totally sure that difference and reason why dart:io WebSocket connections break with error is that dart:io header keys are converts to lowercase.

Successful communication looks like:

GET /app/%3Ftransient%3D HTTP/1.1
Connection: Upgrade
Upgrade: websocket
Host: 192.168.188.10
Sec-WebSocket-Version: 13
Sec-WebSocket-Key: MTMtMTQ0OTMzNjYzNzY0NQ==
Content-Type: application/json
Cookie: X-Qlik-Session=70a54c5b-a08b-4665-a0d7-1a39871ebea9; Path=/; HttpOnly; Secure
Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits

HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: Tb5mxSaTiFApEyYAAv/ci7AMOzI=
Access-Control-Allow-Origin: ws://192.168.188.10

Unsuccessful communication looks like:

GET /app/%3Ftransient%3D HTTP/1.1
user-agent: Dart/1.14 (dart:io)
connection: Upgrade
cache-control: no-cache
accept-encoding: gzip
cookie: X-Qlik-Session=70a54c5b-a08b-4665-a0d7-1a39871ebea9; Path=/; HttpOnly; Secure
content-length: 0
sec-websocket-version: 13
host: 192.168.188.10
sec-websocket-extensions: permessage-deflate; client_max_window_bits
content-type: application/json
sec-websocket-key: lVH5YxHL8QjS7ddjIJ/Ohg==
upgrade: websocket

HTTP/1.1 404 Not Found
Content-Length: 0
Access-Control-Allow-Origin: ws://192.168.188.10

I did not find any way to pass intact (not converted to lowercase) headers to dart:io WebSocket or HttpClient. So finally I've tried raw Socket and with it (writing to it capitalized Connection and Upgrade as header keys) I've managed to get switching protocol response from the Qlik Sense WebSocket server. So apparently from that I can proceed to create WebSocket from manually upgraded WebSocket.

But I wonder now- is Qlik Sense WebSocket server implementation is wrong to reject dart:io client WebSocket connections? Or dart:io WebSockets are somehow not strictly adhere to specification?
Dart:io WebSocket client successfully connects to http://www.websocket.org/ sample server, to dart:io WebSocket servers, to nodejs ws server - but how many Qlik Sense like not compatible with dart:io WebSocket servers are in the wild?

Communication example on Wikipedia looks more like nodejs version (with capitalized keys)

GET /chat HTTP/1.1
Host: server.example.com
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: x3JJHMbDL1EzLkh9GBhXDw==
Sec-WebSocket-Protocol: chat, superchat
Sec-WebSocket-Version: 13
Origin: http://example.com


GET /app/%3Ftransient%3D HTTP/1.1
Connection: Upgrade
Upgrade: websocket
Host: 192.168.188.10
Sec-WebSocket-Version: 13
Sec-WebSocket-Key: MTMtMTQ0OTMzNjYzNzY0NQ==
Content-Type: application/json
Cookie: X-Qlik-Session=70a54c5b-a08b-4665-a0d7-1a39871ebea9; Path=/; HttpOnly; Secure
Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits

HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: Tb5mxSaTiFApEyYAAv/ci7AMOzI=
Access-Control-Allow-Origin: ws://192.168.188.10

Unsuccessful communication looks like:

GET /app/%3Ftransient%3D HTTP/1.1
user-agent: Dart/1.14 (dart:io)
connection: Upgrade
cache-control: no-cache
accept-encoding: gzip
cookie: X-Qlik-Session=70a54c5b-a08b-4665-a0d7-1a39871ebea9; Path=/; HttpOnly; Secure
content-length: 0
sec-websocket-version: 13
host: 192.168.188.10
sec-websocket-extensions: permessage-deflate; client_max_window_bits
content-type: application/json
sec-websocket-key: lVH5YxHL8QjS7ddjIJ/Ohg==
upgrade: websocket

HTTP/1.1 404 Not Found
Content-Length: 0
Access-Control-Allow-Origin: ws://192.168.188.10

I did not find any way to pass intact (not converted to lowercase) headers to dart:io WebSocket or HttpClient. So finally I've tried raw Socket and with it (writing to it capitalized Connection and Upgrade as header keys) I've managed to get switching protocol response from the Qlik Sense WebSocket server. So apparently from that I can proceed to create WebSocket from manually upgraded WebSocket.

But I wonder now- is Qlik Sense WebSocket server implementation is wrong to reject dart:io client WebSocket connections? Or dart:io WebSockets are somehow not strictly adhere to specification?
Dart:io WebSocket client successfully connects to http://www.websocket.org/ sample server, to dart:io WebSocket servers, to nodejs ws server - but how many Qlik Sense like not compatible with dart:io WebSocket servers are in the wild?

Communication example on Wikipedia looks more like nodejs version (with capitalized keys)

GET /chat HTTP/1.1
Host: server.example.com
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: x3JJHMbDL1EzLkh9GBhXDw==
Sec-WebSocket-Protocol: chat, superchat
Sec-WebSocket-Version: 13
Origin: http://example.com

[sgjesse]

Thanks for the report, however the current behaviour is by design.

The HTTP/1.1 specification section 4.2 clearly states that the field names in HTTP headers are case-insensitive. Early on in the design of the Dart HTTP library we decided to make all header fields lowercase, and not try to neither keep the passed-in case nor produce the Xxx-Yyyy-Zzz format.

Btw. the HTTP/2 specification went with all lowercase headers.

[mike-mathieu]

Has anyone found a workaround for this on the client side as I've ran into the same issue?

[zoechi]

@bigmikehoncho I think you should try to get the server fixed. Have you tried to report it as a bug?

[mike-mathieu]

thanks @zoechi they were able to change their protocol.

[Murtuzakabul]

Is there a way in dart to override this default behavior. It is possible to pass the headers but they too are getting converted to lowercase.

[Murtuzakabul]

I am having the same issue with kestrel web server. Discussed the same with kestrel team but not sure what does not adhere to standard. However, I would really appreciate if dart does not try to convert the headers passed by the user. Let the developer decide if she wants to have the headers in lowercase or uppercase.

Also, I suggest, if any header is explicitly provided, it should override the similar generated header. This way, the dart websockets can offer higher compatibility with different servers. There should also be some way to suppress any specific header.

[zoechi]

@Murtuzakabul don't expect this to be changed in Dart.
If the kestrel server depends on casing of headers it's a bug. The HTTP specs explicitly state that headers need to be treated case-insensitive.

[Murtuzakabul]

@zoechi I did some research and it turned out that the lower casing is not the problem. The problem is content-length header. If I remove the content-length header, kestrel would happily accept the request.

This comes to a general question, is there any way to suppress a header (as there is already a way to add additional headers) in Dart ?

It is quite clear that content-length header is not required to negotiate websocket.

[zoechi]

@Murtuzakabul I think you should create a new issue for that. This one is long closed.

[Murtuzakabul]

@zoechi Okay, going to create a new one

[klonk]

@vadimtsushko Can you please provide raw Socket example ? Im facing same problem with signalR connection ( oddly few time i created connection ) .

Thank you

[Ali-Azmoud]

@whesse

You solution is not going to work, I don't how you find the answer while its clearly mentioned in the documentations that

If headers is provided, there are a number of headers which are controlled by the WebSocket connection process. These headers are:

connection
sec-websocket-key
sec-websocket-protocol
sec-websocket-version
upgrade

If any of these are passed in the headers map they will be ignored.

so how did you change these values while Websocket ignore them ?

I am facing this error

Connection to 'http://127.0.0.1:10000/chat#' was not upgraded to websocket

I almost tried every possible way to fix it. port forwarding, using 10.0.2.2:10000, using NO_PROXY as an environment key and so on. but none of them worked

I use the same app in flutter docs to establish a websocket and it connects to ws://echo.websocket.org but I don't know why the hell it does not connect to my localhost nodejs websocket

[csells]

@zoechi Okay, going to create a new one

@Murtuzakabul Did you ever create the issue related to the extra content-length header?

[shaxxx]

Since I can't say to thousands of users to update their receivers or fix their HTTP server implementation, and I don't expect Dart team to change this behavior, I had no other choice but to change default HTTP client.

I took current HttpClient from master branch (requires Dart 2.5, Flutter 1.8+), changed it to leave headers alone and packed in the package. Hopefully it will save someone someone from crying and pulling up their hair. Just use it as a drop in replacement for HttpClient. Tested and works with Dio.

You can find package HERE.

[box-software]

I'm receiving this error when I try to use graphql_flutter with NestJS/Express server. graphql_flutter has its own internal implementation and I can't replace HttpClient as you suggested. I'm wondering if there's any alternative.

[qguv]

I'd just like to point out that, according to the WebSocket specification, the 'S' in 'WebSocket' is to be capitalized. This does seem to conflict with the HTTP specification, but it's there; and some client and server implementations in the wild are so picky one way or another about the casing that it's not possible to get certain servers to talk to certain clients. ¯\_(ツ)_/¯