Permessage-deflate extension #144

SirCipher · 2020-09-24T13:39:27Z

This PR adds support for permessage-deflate for both inbound and outbound messages through the use of WebSocket extensions.
I've also added this support to the tokio-tungstenite crate in a separate PR.

This is feature-gated through the deflate flag and uses the Flate2 crate for inflating and deflating messages.

I've updated the client and server Autobahn results and all unit tests for both pass.

Permessage-deflate

Merge base

Outbound message chunking

TX message chunking

daniel-abramov

First of all: thanks for the PR and the changes that you made!

I left some comments, which I think must be addressed before we can merge them. I also have not yet checked deflate.rs implementation.

Also I have one general question: do we really need an additional type parameter in all places? Maybe we can avoid it? Having a generic trait for extensions is fine, but can't we use many extensions and is a Config a right place to choose the extension? I'm not sure at the moment. It seems like the current implementation assumes that encoder is just a concrete implementation of permessage-deflate, rather than a generic extension or list of extensions.

@agalakhov could you please check the comments that I left to this PR and check if I missed something? This is quite a huge PR and I would check it carefully before we can merge it. But I think there is some work that has to be done.

daniel-abramov · 2020-10-19T17:11:08Z

Cargo.toml

+optional = true
+version = "1.0"
+default-features = false
+features = ["zlib"]


BTW, do we want to rely on zlib? @agalakhov, what do you think?

daniel-abramov · 2020-10-19T17:11:20Z

README.md

@@ -58,7 +58,7 @@ Features
 Tungstenite provides a complete implementation of the WebSocket specification.
 TLS is supported on all platforms using native-tls.

-There is no support for permessage-deflate at the moment. It's planned.
+Permessage-deflate.


Perhaps we can remove this line now ;)

src/protocol/mod.rs

daniel-abramov · 2020-10-19T17:26:46Z

src/protocol/mod.rs

-                if hdr.rsv1 || hdr.rsv2 || hdr.rsv3 {
-                    return Err(Error::Protocol("Reserved bits are non-zero".into()));
-                }
-            }


Does it have any adverse effects that we postpone this check?

If the deflate extension is disabled, then it delegates the work to the uncompressed extension and does not mutate the frame in any way.

src/handshake/client.rs

src/handshake/server.rs

daniel-abramov · 2020-10-19T17:41:48Z

src/protocol/mod.rs

    /// Create a WebSocket context that manages a post-handshake stream.
-    pub fn new(role: Role, config: Option<WebSocketConfig>) -> Self {
+    pub fn new(role: Role, config: Option<WebSocketConfig<Ext>>) -> Self {
+        let config = config.unwrap_or_else(Default::default);


Out of curiosity: why did you decide to move out the config creation line?

src/extensions/uncompressed.rs

daniel-abramov · 2020-10-19T17:48:40Z

src/extensions/deflate.rs

@@ -0,0 +1,922 @@
+//! Permessage-deflate extension


Did you write this file manually? Or is it a modified copy-paste from deflate? I'm asking, because it's quite a huge file and I and @agalakhov will have to validate it carefully as it contains loops, unwraps and some copy-pasting.

It's an adaptation of WS-RS's deflate extension to use ZLib instead of libc, and to work with a growable fragment buffer.

Got it. I think I or @agalakhov will have to take a close look at it. At the first glance, it seems like it contains some copy-paste that we would like to avoid :)

SirCipher · 2020-10-20T09:18:58Z

Hi @application-developer-DA, thank you for the review. Regarding the additional type parameter, the thought behind this was that it allowed for users to implement their own extensions and provide them. An alternative approach to this would be to have permessage-deflate configurable in the WebSocketConfig structure and then select the appropriate encoder when building the WebSocket. If this is something that you would prefer, then I'm happy to make the change, and then that would resolve a number of the comments that you have made.

Regarding your error handling/into comments, the ? operator doesn't work with the trait's associated type for the error - which is why I matched on the error and then did Into::into, though this could be tidied up with map_err.

daniel-abramov · 2020-10-27T17:20:01Z

@SirCipher , yes, I think having it inside a config makes sense. It seems like it would simplify many things. What do you think?

…o deflate-merge � Conflicts: � src/protocol/mod.rs

# Conflicts: # examples/autobahn-client.rs # examples/autobahn-server.rs # src/client.rs # src/handshake/client.rs # src/handshake/server.rs # src/lib.rs # src/protocol/frame/frame.rs # src/protocol/frame/mod.rs # src/protocol/mod.rs # tests/connection_reset.rs

novacrazy · 2021-01-21T03:25:59Z

What's the status of this PR?

daniel-abramov · 2021-01-29T20:52:45Z

@novacrazy the PR does not pass all of our CI/CD checks and has some conflicts, thus it can't be merged. I'm going to look at the content one more time to see if the rest is fine.

daniel-abramov

Thanks for doing that great progress, but unfortunately there are several reasons why we can't merge it right now:

The PR does not pass our current CI/CD pipeline.
It looks like part of this PR should be done as a separate library, i.e. I think that the whole deflate/compression/uncompression logic here should not be part of tungstenite-rs as it could be generalized and does not have anything to do with WebSockets. This would make the implementation much more simple and concise. I'm afraid that if we merge it "as is" (even if (1) is fixed), we would introduce some technical debt to the library which we would like to avoid if possible, unless I'm missing on something.

SirCipher · 2021-02-19T09:55:52Z

Due to time constraints on my end and the additional changes required to make the requested changes, I won't be able to finish this PR in the near future.

SirCipher and others added 24 commits September 5, 2020 00:00

Adds initial deflate implementation

e93f9a0

Updates compression

b08061a

Removes autobahn output folders

1ddb454

Fixes compression truncation issue

e24c00d

Refactors deflate extension handling

0127b55

Updates feature flags

cd49fc2

Adds extension header parameter parsing

8dccd2f

Adds client complete permessage-deflate implementation

d6f4954

Integrates permessage-deflate into servers

77ffd34

Fixes failing unit tests

0d597fa

Tidy up

0c8ae53

Feature-gates flate2

2be3d31

Resolves PR comments

bd170a0

Refactors deflate extension

15621a0

Tidy up

592ee86

Updates autobahn server example

941b3f8

Resets fragments length after fragment buffer is reset

4714182

Adds Default implementation for UncompressedExt

31b50bf

Adds support for outbound message chunking

3cf0b83

Reformat

2744d1b

Renames ambiguous extension type parameters

fcf56c3

Changes chunk iterator to use field for chunk size

710be47

Merge pull request #1 from SirCipher/master

cd43267

Permessage-deflate

Merge pull request #3 from snapview/master

42e6fd8

Merge base

SirCipher mentioned this pull request Sep 24, 2020

Permessage-deflate snapview/tokio-tungstenite#120

Closed

SirCipher added 2 commits October 6, 2020 12:34

Merge pull request #2 from SirCipher/chunked

8b809f7

Outbound message chunking

Merge pull request #4 from swimos/chunked

e71f19b

TX message chunking

daniel-abramov requested changes Oct 19, 2020

View reviewed changes

SirCipher added 9 commits November 2, 2020 15:41

Refactors extension handling

13cfb4d

Refactors extension handling

7795ca1

Splits client/server max_window_bits

b658064

Splits up websocket frame and extension headers

672572e

Updates request header parsing

7795147

Updates documentation

fd43b07

Fixes decompression

31d05fb

Splits client and server deflate config

e0cadce

Merge branch 'master' of https://github.com/swimos/tungstenite-rs int…

a282200

…o deflate-merge � Conflicts: � src/protocol/mod.rs

tinaun mentioned this pull request Nov 24, 2020

Support for permessage-deflate [✅/awaiting_dependency_merge] #2

Open

SirCipher added 5 commits December 1, 2020 09:58

Updates unit tests to check the correct headers

24b7af1

Resolves merge issues

21aac8e

Updates tests

2dab651

Tidy up

a7f0702

SirCipher requested a review from daniel-abramov December 1, 2020 11:10

SirCipher mentioned this pull request Dec 1, 2020

Per-message deflate extension snapview/tokio-tungstenite#139

Closed

jxs mentioned this pull request Jan 12, 2021

WebSocket permessage-deflate extension support seanmonstar/warp#770

Open

daniel-abramov requested changes Jan 29, 2021

View reviewed changes

SirCipher closed this Feb 19, 2021

daniel-abramov mentioned this pull request Mar 4, 2021

Is there any way to support compression ? #187

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Permessage-deflate extension #144

Permessage-deflate extension #144

SirCipher commented Sep 24, 2020

daniel-abramov left a comment

daniel-abramov Oct 19, 2020

daniel-abramov Oct 19, 2020

daniel-abramov Oct 19, 2020

SirCipher Oct 20, 2020

daniel-abramov Oct 19, 2020

daniel-abramov Oct 19, 2020

SirCipher Oct 20, 2020

daniel-abramov Oct 27, 2020

SirCipher commented Oct 20, 2020

daniel-abramov commented Oct 27, 2020

novacrazy commented Jan 21, 2021

daniel-abramov commented Jan 29, 2021

daniel-abramov left a comment

SirCipher commented Feb 19, 2021

Permessage-deflate extension #144

Permessage-deflate extension #144

Conversation

SirCipher commented Sep 24, 2020

daniel-abramov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SirCipher commented Oct 20, 2020

daniel-abramov commented Oct 27, 2020

novacrazy commented Jan 21, 2021

daniel-abramov commented Jan 29, 2021

daniel-abramov left a comment

Choose a reason for hiding this comment

SirCipher commented Feb 19, 2021