New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Decode failed using akka.http.scaladsl.coding.Deflate #1359

Closed
huangbiubiu opened this Issue Aug 16, 2017 · 4 comments

Comments

Projects
None yet
3 participants
@huangbiubiu

huangbiubiu commented Aug 16, 2017

When I try to use akka.http.scaladsl.coding.Deflate to decode HTTP response, there was an exception: akka.stream.impl.io.ByteStringParser$ParsingException: Parsing failed in step akka.http.scaladsl.coding.DeflateDecompressor$$anon$1$$anon$2@1bb42e8 . I believe I receive correct bytes, so I try to use java.util.zip.InflaterInputStream to decode the HTTP entity with deflate encode, and it works.

This is Akka version (will cause an exception):

     
      responseFuture
            .map(Deflate.decodeMessage(_))
            .map(_.entity)
            .flatMap(_.toStrict(1 seconds)(materializer))
            .map(_.data)
            .map(_.utf8String)//this will throw a parse error exception

and this is Java version:

responseFuture
            .map(_.entity)
            .flatMap(_.toStrict(1 seconds)(materializer))
            .map(_.dataBytes)
            .map(_.runWith(StreamConverters.asInputStream()))
            .map(new InflaterInputStream(_, new Inflater(true)))
            .map(Source.fromInputStream(_).mkString)//this should be fine

The entire source code can be seen in Github gist

@jrudolph

This comment has been minimized.

Show comment
Hide comment
@jrudolph

jrudolph Aug 16, 2017

Member

According to https://tools.ietf.org/html/rfc7230#section-4.2.2:

   The "deflate" coding is a "zlib" data format [RFC1950] containing a
   "deflate" compressed data stream [RFC1951] that uses a combination of
   the Lempel-Ziv (LZ77) compression algorithm and Huffman coding.

      Note: Some non-conformant implementations send the "deflate"
      compressed data without the zlib wrapper.

It seems the server you run against is missing the wrapping, so you need to use Inflater(nowrap = true) while akka-http's Deflate implementation uses Inflater(nowrap = false) according to the spec.

There might be a way to guess if the zlib wrapper exists or not, if that's the case, we would be open to a PR that adds auto detection.

Member

jrudolph commented Aug 16, 2017

According to https://tools.ietf.org/html/rfc7230#section-4.2.2:

   The "deflate" coding is a "zlib" data format [RFC1950] containing a
   "deflate" compressed data stream [RFC1951] that uses a combination of
   the Lempel-Ziv (LZ77) compression algorithm and Huffman coding.

      Note: Some non-conformant implementations send the "deflate"
      compressed data without the zlib wrapper.

It seems the server you run against is missing the wrapping, so you need to use Inflater(nowrap = true) while akka-http's Deflate implementation uses Inflater(nowrap = false) according to the spec.

There might be a way to guess if the zlib wrapper exists or not, if that's the case, we would be open to a PR that adds auto detection.

@huangbiubiu

This comment has been minimized.

Show comment
Hide comment
@huangbiubiu

huangbiubiu Aug 30, 2017

@jrudolph can Deflate.decodeMessage() add a optional parameter like decodeMessage(nowarp : Boolean = false) to make it work in this case?

huangbiubiu commented Aug 30, 2017

@jrudolph can Deflate.decodeMessage() add a optional parameter like decodeMessage(nowarp : Boolean = false) to make it work in this case?

@raboof

This comment has been minimized.

Show comment
Hide comment
@raboof

raboof Aug 30, 2017

Member

That sounds neat to me (it'd probably have to be a second method for binary compatibility) - would you be up to creating a PR?

Member

raboof commented Aug 30, 2017

That sounds neat to me (it'd probably have to be a second method for binary compatibility) - would you be up to creating a PR?

tom-walford added a commit to tom-walford/akka-http that referenced this issue Nov 8, 2017

Decode failed using akka.http.scaladsl.coding.Deflate akka#1359
- Added support in Deflate for decoding a message overriding the noWrap setting on the Inflator

@jrudolph jrudolph closed this in f371a7c Nov 28, 2017

@jrudolph jrudolph added this to the 10.0.11 milestone Nov 28, 2017

@jrudolph

This comment has been minimized.

Show comment
Hide comment
@jrudolph

jrudolph Nov 28, 2017

Member

Fixed by #1522.

Member

jrudolph commented Nov 28, 2017

Fixed by #1522.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment