Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add transparent Content-Encoding: gzip/deflate support #155

Open
seriyps opened this issue Jan 19, 2015 · 25 comments

Comments

Projects
@seriyps
Copy link

commented Jan 19, 2015

So, should add Accept-Encoding: gzip,deflate to request, then check for Content-Encoding: gzip or Content-Encoding: deflate and decompress body (after de-chunking).
Decompression may be done in single pass (zlib:gunzip/1 for gzip and zlib:unzip/1 for deflate) in case of synchronous requests and using streaming (zlib:open/0, zlib:inflateInit/1, zlib:inflate/2 ..., zlib:close/1) in case of asynchronous.
Single-pass example can be found here https://github.com/seriyps/xhttpc/blob/master/src/middlewares/compression_middleware.erl

Some option should be added to enable this feature.

@benoitc benoitc added the enhancement label Feb 6, 2015

@benoitc

This comment has been minimized.

Copy link
Owner

commented Feb 6, 2015

will add it in next release. Thanks for the suggestion.

@zweizeichen

This comment has been minimized.

Copy link

commented Feb 14, 2015

+1

@seriyps

This comment has been minimized.

Copy link
Author

commented Mar 27, 2015

Also, bear in mind new Erlang 18.0 API to protect from gzip bombs http://www.erlang.org/documentation/doc-7.0-rc1/erts-7.0/doc/html/zlib.html#inflateChunk-2

@benoitc

This comment has been minimized.

Copy link
Owner

commented Mar 27, 2015

feature is coming. I am in the middle of adding some big improvments to hackney. I will probably include it right after. Thanks for the hint.

@tuscland

This comment has been minimized.

Copy link
Contributor

commented Jul 22, 2015

+1 :)

@zyro

This comment has been minimized.

Copy link
Contributor

commented Aug 22, 2015

+1 Would be great to see this handled transparently for both requests and responses.

@benoitc benoitc added this to the 2.0.0 milestone Aug 25, 2015

@huoshiqiu

This comment has been minimized.

Copy link

commented Nov 11, 2015

+1 :)

@Anonyfox

This comment has been minimized.

Copy link

commented Dec 18, 2015

+1 on this :)

@kostya

This comment has been minimized.

Copy link

commented Jan 20, 2016

+1

@edgurgel

This comment has been minimized.

Copy link
Collaborator

commented Apr 11, 2016

Can I help with this? I would be glad to follow some guidance and help with this feature.

@benoitc

This comment has been minimized.

Copy link
Owner

commented Apr 14, 2016

@edgurgel thanks! It should probably be added to the function receiving the body, if you wrap them and keep the state it should be OK. Anyway betetr to wait over the we, with the new recv and recv_multipart functions :)

@tlvenn

This comment has been minimized.

Copy link

commented Nov 21, 2016

Any update on this feature ?

@loicvigneron

This comment has been minimized.

Copy link

commented Nov 29, 2016

+1 :)

@stevegraham

This comment has been minimized.

Copy link

commented Feb 10, 2017

What's the status of this issue?

@benoitc

This comment has been minimized.

Copy link
Owner

commented Feb 10, 2017

No work have been done on my side yet. But any help is appreciated. Anyway I expect that this feature will land after the current rework of the internals.

@sunboshan

This comment has been minimized.

Copy link

commented Feb 24, 2017

+1. Just encountered this issue that some server send gzip content and Hackey returns bitstring in body. Finally figure the root issue with lots of people's help. Though it would be really good if Hackey can transparently unzip the body based on Content-Encoding. :)

@benoitc benoitc added http11 http and removed http11 labels May 19, 2017

@benoitc benoitc added this to HTTP in Development May 19, 2017

@jimdigriz

This comment has been minimized.

Copy link
Contributor

commented May 21, 2017

I have a shim wrapper for this that works for default/gzip and even when calling stream_body/1 that I can put up here if or anyone wants to see the moving parts that are involved? It is not a hackney fork though.

@edevil

This comment has been minimized.

Copy link

commented Aug 2, 2017

Have the internals been reworked?

@cgorshing

This comment has been minimized.

Copy link

commented Oct 29, 2017

I'm working on a ueberauth strategy for StackOverflow and I would love to have this feature completed so I don't have to wire it into OAuth2's handling of the response.

I have some time available to actively help develop, test, or otherwise pitch in. How might I be able to lend a hand? Is a branch started for this that I can begin looking at?

@jimdigriz

This comment has been minimized.

Copy link
Contributor

commented Oct 29, 2017

@cgorshing

This comment has been minimized.

Copy link

commented Oct 29, 2017

Responses from StackOverflow are gzipped. Ueberauth hands off the request to the OAuth2 library, that just calls Poison for the deserialization of the body. The OAuth2 library does not pass the headers to Poison, just the body.

I'm getting around this by creating my own serializer to gunzip the body before handing off to Poison (https://github.com/scrogson/oauth2/blob/master/lib/oauth2/serializer.ex#L27-L39). But in doing this, I still don't have access to the headers so it's blind faith right now.

Hit me up on Elixir's slack if you are more interested, I feel like this is a tad off-topic for this issue.

@jimdigriz

This comment has been minimized.

Copy link
Contributor

commented Oct 30, 2017

Surely only when you include a accept-encoding: gzip, deflate header though?

@fenollp

This comment has been minimized.

Copy link

commented Nov 11, 2017

@jimdigriz Can you share your shim with us?

@jimdigriz

This comment has been minimized.

Copy link
Contributor

commented Nov 12, 2017

Okay...it is ugly as sin though. Attached is code you will just have to adapt to fit into your setup.

zlib-shim.txt - lets call this the http module

So instead really what you want is:

  • hackney:get -> http:get (you cannot use the get body option here)
  • hackney:body -> http:body
  • hackney:stream_body -> http:stream_body
  • http:stream_body_drain is used for when you need to drain and discard a request; though probably show not exist as you could just call http:body...
@jimdigriz

This comment has been minimized.

Copy link
Contributor

commented Nov 28, 2017

Anyone tried #456, it has been in production for my use case for two weeks?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.