Content-Encoding: deflate is actually zlib

[sschepens]

Had this issue with another library.
It's a little counter-intuitive, but Content-Encoding: deflate is actually not a simple flate stream, but a zlib one.
According to [Wikipedia](https://en.wikipedia.org/wiki/HTTP_compression:

deflate – compression based on the deflate algorithm (described in RFC 1951), wrapped inside the zlib data format (RFC 1950);

fasthttp currently uses flate package for handling deflated content, but should use zlib

Also, it would be great if fasthttp used: https://github.com/klauspost/compress as it provides optimized compression packages

[valyala]

@sschepens , could you confirm that fasthttp now properly handles Content-Encoding: deflate?

[valyala]

As for @klauspost 's compress library, I'll look into it later.

[klauspost]

AFAIR there are different opinions on what deflate means, which is why Content-Encoding: gzip is generally preferred.

Found a some information from Mark Adler:

[]...
However early Microsoft servers would incorrectly deliver raw deflate for "Deflate" (i.e. just RFC 1951 data without the zlib RFC 1950 wrapper). This caused problems, browsers had to try it both ways, and in the end it was simply more reliable to only use gzip.

The impact in bandwidth and execution time to use gzip instead of "Deflate" (zlib), is relatively small. So there we are and there it is likely to remain.

The difference is 12 more bytes for gzip and slightly more CPU time to calculate a CRC instead of an Adler-32.

CRC will actually be faster on AMD64, since there are assembler optimized routines (if you use my compression package), so the last point isn't valid for Go.

[sschepens]

@klauspost I know gzip is usually preferred over deflate, but I use some services that only support deflate. I believe nowadays it's more common to have deflate as zlib than raw deflate, and according to standard should be zlib.

[valyala]

@klauspost , are there chances for the integration of your compression performance improvements into standard Go library?

[klauspost]

@valyala - yes, I have just begun submitting the CLs, but it will of course not be out until 1.7.

Meanwhile, it can easily be integrated. There should be nothing more than an import change to use the current code, which is roughly 1.7x faster than the built-in, with no allocations if the Encoder is reused - which you appear to already do.

[valyala]

Substituted standard compress library with https://github.com/klauspost/compress

[sschepens]

👍