Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

compress/flate: Allow resetting writer with new dictionary #36919

Open
nhooyr opened this issue Jan 31, 2020 · 14 comments
Open

compress/flate: Allow resetting writer with new dictionary #36919

nhooyr opened this issue Jan 31, 2020 · 14 comments

Comments

@nhooyr
Copy link
Contributor

@nhooyr nhooyr commented Jan 31, 2020

As of Go 1.14, compress/flate's Writer only allows resetting the write side with the same dictionary.

In contrast, the Reader can be reset with a new dictionary.

I need this to efficiently implement context takeover compression for WebSockets.

See https://tools.ietf.org/html/rfc7692#section-7.1.1

cc @klauspost

@klauspost
Copy link
Contributor

@klauspost klauspost commented Jan 31, 2020

What is the problem you are trying to solve? Is this a 1.14 change?

I'm not sure I understand why it is a problem to supply the dictionary when resetting?

Loading

@as
Copy link
Contributor

@as as commented Jan 31, 2020

Possibly related #18930

Loading

@nhooyr
Copy link
Contributor Author

@nhooyr nhooyr commented Jan 31, 2020

@klauspost

I don't want to keep a flate.Writer allocated for every connection due to memory usage. Instead, I'd rather store the rolling window myself and then grab a writer out of the pool and reset it with the window and then write a message to the connection.

Loading

@nhooyr
Copy link
Contributor Author

@nhooyr nhooyr commented Jan 31, 2020

I'm not sure I understand why it is a problem to supply the dictionary when resetting?

It's not a problem, that's exactly what I want to do but it's unsupported at the moment.

Loading

@klauspost
Copy link
Contributor

@klauspost klauspost commented Jan 31, 2020

Ah, ok, so like a Stateless compression function that accepts a dictionary or a Stateless Writer, where it keeps the history between calls, but discards everything else.

Loading

@nhooyr
Copy link
Contributor Author

@nhooyr nhooyr commented Jan 31, 2020

Ah my bad I forgot you already implemented that.

Is there an issue open to merge it into the stdlib or shall I keep this open?

Loading

@nhooyr
Copy link
Contributor Author

@nhooyr nhooyr commented Jan 31, 2020

Nvm, there is no dict option yet in your pkg.

Loading

@klauspost
Copy link
Contributor

@klauspost klauspost commented Jan 31, 2020

I don't have plans to submit it to the stdlib unless someone expresses a need for it. The threshold for adding stuff is of course a bit higher for the stdlib.

Adding it to the functions above would not be super easy, since everything related to history was pulled out to make it stateless. OTOH is would also not be massive. Having, say a max 8K dict would be feasible, but every call would of course require it to be re-indexed, so not free.

Loading

@klauspost
Copy link
Contributor

@klauspost klauspost commented Feb 1, 2020

@nhooyr You can test klauspost/compress#216

Slightly clunky API to remain backwards compatible.

Loading

@nhooyr
Copy link
Contributor Author

@nhooyr nhooyr commented Feb 1, 2020

Will do this weekend, thanks @klauspost

Loading

@nhooyr
Copy link
Contributor Author

@nhooyr nhooyr commented Feb 2, 2020

Won't be able to get to it this weekend.

Appreciate your very quick response to this issue @klauspost, I'll try to get to get to it next week.

Loading

@nhooyr
Copy link
Contributor Author

@nhooyr nhooyr commented Apr 14, 2020

Using this in my WebSocket library and the performance has been great.

klauspost/compress#216 (comment)

2x faster and only 100 B allocated per op for echoing a 512 random byte message with a 8 KB dictionary.

Loading

@nhooyr
Copy link
Contributor Author

@nhooyr nhooyr commented Apr 14, 2020

Would be nice to see it in the stdlib instead to avoid the dependency.

Loading

@nhooyr
Copy link
Contributor Author

@nhooyr nhooyr commented Apr 14, 2020

Adding it would close #32371 as well.

Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants