Implement WebSocket compression

[Tratcher]

https://tools.ietf.org/html/rfc7692

Autobahn test section 12

[muratg]

@Tratcher let's discuss this (and other WebSockets work) when you're back.

[kevygreen]

This would be a great addition. Especially when you are using SignalR as this would help keep the communication to the browser much smaller when sending repetitive $type information. It would be nice if this could be done as part of the core instead of trying to hack it in as an add-on after the fact if it's even possible at the higher levels without ripping out all the base levels.

[deandob]

Any progress on this? I have a server app that has a dependency on websockets compression and none of the libraries that support compression are ported to Core so its blocking me porting the whole app to Core. Would be great to have native websockets compression support so that no additional libraries are needed.

[Tratcher]

No progress to report, so far you two are the only ones that have asked for it.

[deandob]

Hi,
Are there any plans for it? Any reasonable websockets payload over the internet would get a good speed boost with compression (and its a standard feature on other websockets stacks)?

Also its not clear what to do about client websockets - an earlier issue just closed about the client mentions to use 'ClientWebSocket' from System.Net, are there plans for a Core version?

[Tratcher]

No plans for it this milestone, it's a large feature and there's been minimal interest.

Yes, ClientWebSocket will be available for Core.
https://github.com/dotnet/corefx/blob/561050c311a09faef1818765baa1159b54371f0a/src/System.Net.WebSockets.Client/src/System/Net/WebSockets/ClientWebSocket.cs

[kevygreen]

I think there is minimal knowledge not necessarily interest. I don't think many people are aware of this level of detail of the spec. They might not even know of the possible gains they could get with such a feature.

We use SignalR as our primary push notification system. With JSON serialization of objects there is a huge communications gain by adding in compression, especially for arrays of objects. We implemented our own compression using LZString and a custom serializer to get a reduction in bandwidth of about 90-98% depending on the messages. This is even with base64encoding which cause some loss of efficiency.

Other SignalR users would likely gain from this as well. This means that out of the box you could offer an option that could save a huge amount of bandwidth for any large or even small SignalR user. I think if you keep this audience alone in mind there are likely thousands who would be interested in this gain. If this was implemented at the lower levels and had proper browser support, the overall impact of the compression would be greatly reduced compared to the hacks we have had to put in place or the effort required to write custom serializer to reduce the data load.

If it was implemented with SignalR in mind it might also be possible to save space on the SignalR Message bus if messages were passed to the bus already compressed.

[deandob]

My use case is similar (and likely quite common for websockets), use of JSON between client and server is easy to blow out the package size significantly and permessage-deflate extension support in ASP Core would see some significant speedups for websockets users. Even though Edge does not (yet) support this extension all the other popular browsers do. Given the large focus on Kestrel's raw performance (which is excellent) it would be nice to see the same performance considerations prioritised for websockets use as well.

But I understand it will take time, so thanks for considering it for a future milestone.

[Tratcher]

@davidfowl would SignalR use WS compression?

[Tratcher]

Implementation notes:
The System.Net.WebSockets.WebSocket.SendAsync API is poorly suited to compression/extension. Compression is a per-message flag set on the first frame using the RSV1 bit. SendAsync sends frames and only takes the buffer, message type, an EndOfMessage bool, and a cancellationToken. There's no simple way to indicate if the data is or should be compressed / transformed. Possible solutions:

• Enable compression for all messages, don't provide a per message option
• Add a new overload of SendAsync with a compression / RSV1 flag. (Requires changing the core WebSocket abstraction)
• Add a property to WebSocket bool CompressOutput that can be toggled between messages (Requires changing the core WebSocket abstraction)
• Overload WebSocketMessageType with CompressedBinary and CompressedText. As an enum we may be able to temporarily pass data through here by casting non-standard values. The enum values already do not map directly to the WebSocket frame type values.

ReceiveAsync is easier, it could:

• Decompress the content internally
• Add a flag to WebSocketReceiveResult indicating if the content was compressed.
• Use the same WebSocketMessageType overloads from above indicating if the content is compressed (but only if it does not decompress it internally, don't want to confuse consumers).

The compression extension could not be correctly implemented as wrapper over the current WebSocket API due to the framing requirement. Compression either needs to be built in or the API needs to expose additional frame fields (e.g. RSV1).

I'll consider the extension negotiation separately as the handshake APIs differ in System.Net, System.Web, and Microsoft.AspNetCore.

We also need to verify that the Sec-WebSocket-Extensions header(s) flow well end-to-end. Intermediaries like ANCM have had problems with other optional headers like Sec-WebSocket-Protocol. For that matter IIS and WinHttp would have to implement this extension as well, their APIs don't allow ANCM to proxy opaque websocket frames.

[davidfowl]

@davidfowl would SignalR use WS compression?

Possibly? Do all browsers support it? Does it transparently fallback if it isn't supported client side?

[Tratcher]

The client has to request it, just like subprotocol. Need to research where else it's supported.

[deandob]

FYI. Here is a c# example of an websockets implementation with compression.
https://github.com/vtortola/WebSocketListener/wiki/Deflate-extension

Thanks for looking into it.

[Tratcher]

@deandob interesting. They use a completely different WebSocket API abstraction that's message based and directly exposes the extensibility flags. Then they wrap the base implementation with another that manages the compression. However, they don't have a way to enable/disable compression on a per message basis.

[Tratcher]

aspnet/HttpAbstractions#711

[deandob]

OK thanks. So I guess it is still a ways away then? Need any community help?

[Tratcher]

The object model issues mean that implementing this will require significant effort. That won't happen without clear customer demand. The best thing you can do right now is help gauge interest.

[RyanLamansky]

The suggestion to overload WebSocketMessageType with CompressedBinary and CompressedText is appealing from a usage perspective. It makes the availability of the feature noticeable via Intellisense and naturally works on a per-message basis. It's not as good from an API purist perspective as we're adding two enum items to cover a single flag.

I think a new overload of SendAsync is best. Compression could be part of a something like a new MessageOptions [Flags] enumeration as its first entry (alongside None = 0). Various other per-message on/off switches could easily be added later.

[Tratcher]

Most of this work needs to be done in https://github.com/dotnet/corefx/issues/15430, ASP.NET only implements the handshake.

[RyanLamansky]

I'm currently being impacted by this issue: I'm converting a polling-based process from XHR to a WebSocket and have discovered that although the site is now faster on high-speed connections (latency reduction via data pushes from the server at the exact moment of readiness), it's slower on low-speed connections (data takes 10x more bandwidth).

[sepehr1014]

Any updates on this? This feature would be extremely beneficial.

[Tratcher]

No, this is blocked on https://github.com/dotnet/corefx/issues/15430.

[deandob]

I'm still looking for this feature and with a sizable SaaS app about to go live that would benefit considerably from compression I'm about to jump onto a 3rd party websockets library. I'm surprised how little interest there has been on this as compression would benefit most websockets use-cases.
For those looking for the same thing and can't wait for the dotnet core team, this is what I'm using:
https://github.com/deniszykov/WebSocketListener

[talarari]

Any update on this?

[jjxtra]

Also eager for this, rolled our own base64 compression on top manually, but this is a big hack.