pulp3: fixture_utils.py: avoid automatic encoding #1310
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
fixes #1309
If a test makes use of a fixture server where the test data/file being served is already compressed the client will erroneously decompress the file. The following describes why.
On the client side, the aiohttp Client will automatically decompress response bodies when the Content-Encoding is one of the compressed types, like gzip.
On the server side the aiohttp Server will set Content-Encoding based on mimetype, which for gzip'd files with be 'gzip'.
--web_fileresponse.py:170--
if hdrs.CONTENT_TYPE not in self.headers:
ct, encoding = mimetypes.guess_type(str(filepath))
This will happen regardless of Accept-Encoding in the request header (which might be considered a bug in aiohttp). As you can see in the above code snippet one way to avoid this automatic setting of Content-Encoding is to set the Content-Type in the header, which is what we do here.
With this change in place the Accept-Encoding appears to be respected, presenting the client with the expected outcomes based on the various RFCs. Which ultimately results in a compressed file not being deflated by the aiohttp client. NOTE the Content-Length, Content-Type and Content-Encoding in the following data grabbed from a scenario served by pulp-smash.
--
curl -I -H 'Accept-Encoding: gzip' http://127.0.0.1:$PORT/debian/dists/nosuite/asgard/binary-ppc64/Packages.gz \ && curl -I -H 'Accept-Encoding: identity' http://127.0.0.1:$PORT/debian/dists/nosuite/asgard/binary-ppc64/Packages.gz \ && curl -I -H 'Accept-Encoding: gzip' http://127.0.0.1:$PORT/debian/dists/nosuite/asgard/binary-ppc64/Packages \ && curl -I -H 'Accept-Encoding: identity' http://127.0.0.1:$PORT/debian/dists/nosuite/asgard/binary-ppc64/Packages
HTTP/1.1 200 OK
content-type: application/octet-stream
Etag: "1750f481b8f1ae00-363"
Last-Modified: Wed, 29 Mar 2023 17:38:19 GMT
Content-Length: 867
Accept-Ranges: bytes
Date: Fri, 14 Apr 2023 16:03:49 GMT
Server: Python/3.10 aiohttp/3.8.4
HTTP/1.1 200 OK
content-type: application/octet-stream
Etag: "1750f481b8f1ae00-363"
Last-Modified: Wed, 29 Mar 2023 17:38:19 GMT
Content-Length: 867
Accept-Ranges: bytes
Date: Fri, 14 Apr 2023 16:03:49 GMT
Server: Python/3.10 aiohttp/3.8.4
HTTP/1.1 200 OK
content-type: application/octet-stream
Content-Encoding: gzip
Vary: Accept-Encoding
Etag: "1750f481b8f1ae00-363"
Last-Modified: Wed, 29 Mar 2023 17:38:19 GMT
Content-Length: 867
Accept-Ranges: bytes
Date: Fri, 14 Apr 2023 16:03:49 GMT
Server: Python/3.10 aiohttp/3.8.4
HTTP/1.1 200 OK
content-type: application/octet-stream
Etag: "1750f481b8f1ae00-6d9"
Last-Modified: Wed, 29 Mar 2023 17:38:19 GMT
Content-Length: 1753
Accept-Ranges: bytes
Date: Fri, 14 Apr 2023 16:03:49 GMT
Server: Python/3.10 aiohttp/3.8.4
Note that uncompressed data (Packages) will be compressed during transmission when Accept-Encoding is 'gzip'. This change should not affect tests based off pulp-smash, but if they do break it is more likely the test was working around this issue.