Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add transparent Content-Encoding: gzip/deflate support #155

Open
seriyps opened this issue Jan 19, 2015 · 27 comments
Open

Add transparent Content-Encoding: gzip/deflate support #155

seriyps opened this issue Jan 19, 2015 · 27 comments

Comments

@seriyps
Copy link

@seriyps seriyps commented Jan 19, 2015

So, should add Accept-Encoding: gzip,deflate to request, then check for Content-Encoding: gzip or Content-Encoding: deflate and decompress body (after de-chunking).
Decompression may be done in single pass (zlib:gunzip/1 for gzip and zlib:unzip/1 for deflate) in case of synchronous requests and using streaming (zlib:open/0, zlib:inflateInit/1, zlib:inflate/2 ..., zlib:close/1) in case of asynchronous.
Single-pass example can be found here https://github.com/seriyps/xhttpc/blob/master/src/middlewares/compression_middleware.erl

Some option should be added to enable this feature.

@benoitc
Copy link
Owner

@benoitc benoitc commented Feb 6, 2015

will add it in next release. Thanks for the suggestion.

Loading

@zweizeichen
Copy link

@zweizeichen zweizeichen commented Feb 14, 2015

+1

Loading

@seriyps
Copy link
Author

@seriyps seriyps commented Mar 27, 2015

Also, bear in mind new Erlang 18.0 API to protect from gzip bombs http://www.erlang.org/documentation/doc-7.0-rc1/erts-7.0/doc/html/zlib.html#inflateChunk-2

Loading

@benoitc
Copy link
Owner

@benoitc benoitc commented Mar 27, 2015

feature is coming. I am in the middle of adding some big improvments to hackney. I will probably include it right after. Thanks for the hint.

Loading

@tuscland
Copy link
Contributor

@tuscland tuscland commented Jul 22, 2015

+1 :)

Loading

@zyro
Copy link
Contributor

@zyro zyro commented Aug 22, 2015

+1 Would be great to see this handled transparently for both requests and responses.

Loading

@jedhu0
Copy link

@jedhu0 jedhu0 commented Nov 11, 2015

+1 :)

Loading

@Anonyfox
Copy link

@Anonyfox Anonyfox commented Dec 18, 2015

+1 on this :)

Loading

@kostya
Copy link

@kostya kostya commented Jan 20, 2016

+1

Loading

@edgurgel
Copy link
Collaborator

@edgurgel edgurgel commented Apr 11, 2016

Can I help with this? I would be glad to follow some guidance and help with this feature.

Loading

@benoitc
Copy link
Owner

@benoitc benoitc commented Apr 14, 2016

@edgurgel thanks! It should probably be added to the function receiving the body, if you wrap them and keep the state it should be OK. Anyway betetr to wait over the we, with the new recv and recv_multipart functions :)

Loading

@tlvenn
Copy link

@tlvenn tlvenn commented Nov 21, 2016

Any update on this feature ?

Loading

@loicvigneron
Copy link

@loicvigneron loicvigneron commented Nov 29, 2016

+1 :)

Loading

@stevegraham
Copy link

@stevegraham stevegraham commented Feb 10, 2017

What's the status of this issue?

Loading

@benoitc
Copy link
Owner

@benoitc benoitc commented Feb 10, 2017

No work have been done on my side yet. But any help is appreciated. Anyway I expect that this feature will land after the current rework of the internals.

Loading

@sunboshan
Copy link

@sunboshan sunboshan commented Feb 24, 2017

+1. Just encountered this issue that some server send gzip content and Hackey returns bitstring in body. Finally figure the root issue with lots of people's help. Though it would be really good if Hackey can transparently unzip the body based on Content-Encoding. :)

Loading

@benoitc benoitc added this to HTTP in Development May 19, 2017
@jimdigriz
Copy link
Contributor

@jimdigriz jimdigriz commented May 21, 2017

I have a shim wrapper for this that works for default/gzip and even when calling stream_body/1 that I can put up here if or anyone wants to see the moving parts that are involved? It is not a hackney fork though.

Loading

@edevil
Copy link

@edevil edevil commented Aug 2, 2017

Have the internals been reworked?

Loading

@cgorshing
Copy link

@cgorshing cgorshing commented Oct 29, 2017

I'm working on a ueberauth strategy for StackOverflow and I would love to have this feature completed so I don't have to wire it into OAuth2's handling of the response.

I have some time available to actively help develop, test, or otherwise pitch in. How might I be able to lend a hand? Is a branch started for this that I can begin looking at?

Loading

@jimdigriz
Copy link
Contributor

@jimdigriz jimdigriz commented Oct 29, 2017

Loading

@cgorshing
Copy link

@cgorshing cgorshing commented Oct 29, 2017

Responses from StackOverflow are gzipped. Ueberauth hands off the request to the OAuth2 library, that just calls Poison for the deserialization of the body. The OAuth2 library does not pass the headers to Poison, just the body.

I'm getting around this by creating my own serializer to gunzip the body before handing off to Poison (https://github.com/scrogson/oauth2/blob/master/lib/oauth2/serializer.ex#L27-L39). But in doing this, I still don't have access to the headers so it's blind faith right now.

Hit me up on Elixir's slack if you are more interested, I feel like this is a tad off-topic for this issue.

Loading

@jimdigriz
Copy link
Contributor

@jimdigriz jimdigriz commented Oct 30, 2017

Surely only when you include a accept-encoding: gzip, deflate header though?

Loading

@fenollp
Copy link

@fenollp fenollp commented Nov 11, 2017

@jimdigriz Can you share your shim with us?

Loading

@jimdigriz
Copy link
Contributor

@jimdigriz jimdigriz commented Nov 12, 2017

Okay...it is ugly as sin though. Attached is code you will just have to adapt to fit into your setup.

zlib-shim.txt - lets call this the http module

So instead really what you want is:

  • hackney:get -> http:get (you cannot use the get body option here)
  • hackney:body -> http:body
  • hackney:stream_body -> http:stream_body
  • http:stream_body_drain is used for when you need to drain and discard a request; though probably show not exist as you could just call http:body...

Loading

@jimdigriz
Copy link
Contributor

@jimdigriz jimdigriz commented Nov 28, 2017

Anyone tried #456, it has been in production for my use case for two weeks?

Loading

@tsloughter
Copy link

@tsloughter tsloughter commented Oct 26, 2021

Was curious if this was supposed and found this issue. Is it still planned for 2.0.0?

Loading

@benoitc
Copy link
Owner

@benoitc benoitc commented Oct 28, 2021

i am lagging in release, but yes that will be part of the next release. There is a RC that will land next week including it :)

Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
No open projects
Linked pull requests

Successfully merging a pull request may close this issue.

None yet