aiohttp decompresses 'Content-Encoding: gzip' responses

[JustAnotherArchivist]

Long story (not so) short

aiohttp automatically decompresses responses with a Content-Encoding header. I believe this is incorrect – while decompression may of course be useful, it should not be on by default.

The Content-Encoding header is similar to the Transfer-Encoding one, and the two are often confused. The difference is that TE is about the data transfer, i.e. the encoding for a particular connection and client, whereas CE is about the data itself. CE exists to allow specifying this encoding without losing the MIME type of the actual data.

As an example, an ideal response for a .tar.gz file download would have a Content-Type: application/x-tar header and a Content-Encoding: gzip one (rather than e.g. Content-Type: application/x-gzip without Content-Encoding). When downloading such a file, one would still expect to get the .tar.gz file, not an uncompressed .tar. This is why I think that aiohttp should not automatically decompress on a Content-Encoding header.

Steps to reproduce

Make a request for a file that has been gzipped, and read from the ClientResponse object (e.g. to write it to a file).

Expected behaviour

Get the gzipped file.

Actual behaviour

Returns the decompressed file.

Your environment

All versions are affected (oldest checked: 2.3.10, but presumably much older than that).

Additional notes

curl also used to have this bug in their HTTP/2 implementation. I haven't searched in detail whether it existed in the HTTP/1.1 one as well, but current versions do not decompress such responses.

[thehesiod]

see #1992

[socketpair]

I'm against. Because it predates RFC. I don't see any real case where we should not decompress automatically. The Content-Encoding header is not related to data contents actually. It is connected with a way data is transferred over HTTP.

HTTP servers that serve data that supposed to be used in compressed form, should not set Content-Encoding: gzip.

I'm closing the issue. Please reopen if you don't think so.

[thehesiod]

there is a real case, where you're downloading a gzip file and you want to store the gzip file. I think this is already supported though if you follow the issue I linked to

[JustAnotherArchivist]

@socketpair I'm not sure what you mean by "it predating RFC". Content-Encoding was specified in HTTP/1.1 in 1999 (here), and as explained in the original issue, it is not the same as Transfer-Encoding (here). The latter is for "the way data is transferred over HTTP", but Content-Encoding is about the actual data itself. Roy T. Fielding, one of the authors of RFC 2616, also clarified this a few times (e.g. here). Unfortunately, over the last two decades, the headers have been misused and their use misadvised countless times, which is what causes the current mess. Nevertheless, CE is for the data, TE is for the transfer, and the proper headers for a .tar.gz, which you would expect to be written to disk as a compressed file, would include CE. If you use aiohttp for such a download, you'd instead only get the .tar.

Please note that I'm not against having an option to decompress bodies with a compresseion CE since there absolutely are valid use cases for that. I'm arguing that this decompression should not happen by default.

@thehesiod Without having tested it, according to the comments in your PR #2110, that disables decoding of both content and transfer encoding. While I'm sure there are use cases for that, in my opinion, only content encoding should be disabled.

[socketpair]

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Encoding

The Content-Encoding entity header is used to compress the media-type. When present, its value indicates which encodings were applied to the entity-body. It lets the client know how to decode in order to obtain the media-type referenced by the Content-Type header.

If someone wants to serve .tar.gz, he MUST NOT use "Content-Encoding: gzip".

For example:
https://github.com/aio-libs/aiohttp/archive/master.tar.gz

$ curl -L -v -v --compressed https://github.com/aio-libs/aiohttp/archive/master.tar.gz -o/dev/null 2>&1 | egrep -i 'content-|encoding'
.......
> Accept-Encoding: deflate, gzip
< Content-Security-Policy: default-src 'none'; style-src 'unsafe-inline'; sandbox
< Vary: Authorization,Accept-Encoding
< X-Content-Type-Options: nosniff
< Content-Type: application/x-gzip
< Content-Disposition: attachment; filename=aiohttp-master.tar.gz
< Content-Length: 833426

I would revert #2110 . I don't see any reasons to have such an option. If S3 server (or its settings) is wrong, we should we care?

For example, this is a hack:
https://www.rightbrainnetworks.com/2013/03/21/serving-compressed-gzipped-static-files-from-amazon-s3-or-cloudfront/

Here you are uploading pre-gzipped contents, instruct S3 to add header and then expect a browser to decompress. It works. Everything OK.

But if you want to download gzipped contents (you possibly need it for the S3 protocol) you have to force server not to send headers you set up before. Because these headers are for end-users, not for S3-aware programs.

S3-aware programs should use ?response-content-encoding=identity or so.

https://botocore.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html

[JustAnotherArchivist]

No, serving a .tar.gz with Content-Type: application/x-tar and Content-Encoding: gzip is perfectly valid. Here's a little minimal web server I wrote a couple months ago for testing purposes. Downloading http://127.0.0.1:8080/f.tar.gz with Firefox, wget, or curl correctly produces a compressed file (not a valid tar though because I was too lazy for that).

From section 7.2.1 of RFC 2616 (emphasis mine):

Content-Type specifies the media type of the underlying data. Content-Encoding may be used to indicate any additional content codings applied to the data, usually for the purpose of data compression, that are a property of the requested resource.

Serving it with Content-Type: application/x-gzip, like GitHub does, is not a violation of the specification (nor would serving it as application/octet-stream be), but it means a loss of header information that the body is a tar file.

[socketpair]

@JustAnotherArchivist I've updated the comment, sorry for the long delay.

[JustAnotherArchivist]

I think that S3 usage is okay, actually. The browser decompresses it if the file is used in a <link rel="stylesheet"> tag because it needs the actual text/css data. This is also covered in the RFC in section 14.11: "Typically, the entity-body is stored with this encoding and is only decoded before rendering or analogous usage." The same thing would apply for compressed images embedded in a page. I added an example of this to the same Gist linked above. Requesting http://127.0.0.1:8080/index.html in Firefox brings up a page with a bright-green background, even though the CSS file is served with gzip CE. If you directly download style.css with wget or curl, however, you get the compressed file, as expected.

[socketpair]

Regarding curl, it has --compressed. This option makes it understand content-encoding the same way as a browser. The same for wget , but --compression=auto.

Well. We may argue more and more. Aiohttp has auto_decompress option. If you still speak about changing the default value, I'm still against it.

P.S.

The reason why I reopen the issue: I figured out, that browsers (at least, Firefox) DOES NOT decompress such responses when they download URL (not render).

@asvetlov I don't know what to do. Both points of view have pros and cons. RFC actually is not clear enough.

[danielrobbins]

@socketpair I agree with the original bug reporter, he is correct. This is a bug. You should fix it.

This is a bug impacting Funtoo Linux due to http://cdn.postfix.johnriley.me/mirrors/postfix-release/official/postfix-3.6.1.tar.gz. This is a CDN serving a gzip-compressed tarball. The artifact being served over HTTP is "postfix-3.6.1.tar.gz". Relevant headers are:

Content-Encoding: gzip
Content-Length: 4748754
Content-Type: application/x-tar

aiohttp returns a 21MB file, uncompressed. In our case, getting the exact binary data of the postfix-3.6.1.tar.gz file is essential -- gzip is not just a transport feature (aka Transfer-Encoding) to reduce bandwidth but we are requesting the contents of "the gzip compressed file" and we need it to match perfectly what is on the origin server because we must compute SHA512 digest of this file. So we need the original compressed version -- we can't recompress it because our gzip might use different settings and get a different binary and thus different SHA512.

What you should do by default is not transparently decompress based on Content-Encoding. CE is for data, TE is for transfer. End of story. If you need to support this, you should add a feature for auto-decompression of content-encoded gzip which could be useful in some cases but it should not be the default behavior.

We must respect the distinct meaning of CE and TE, and the concepts of "content" and "transfer". When I download, I want the "content".

Upstream bug: https://bugs.funtoo.org/browse/FL-8477

[danielrobbins]

Looking into this more, it seems like the reality of HTTP servers and their handling of Content-Encoding is not always according to what we would hope, or in alignment with HTTP specs.

A workaround for our specific issue was to specify "Accept-Encoding: identity" in the request, which informed the CDN to send the file as-is with any Content-Encoding (it is already compressed on disk, so this is fine) thus avoiding the issue.