Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pulp3: fixture_utils.py: avoid automatic encoding #1310

Merged
merged 1 commit into from
Apr 18, 2023

Conversation

masselstine
Copy link
Contributor

fixes #1309

If a test makes use of a fixture server where the test data/file being served is already compressed the client will erroneously decompress the file. The following describes why.

On the client side, the aiohttp Client will automatically decompress response bodies when the Content-Encoding is one of the compressed types, like gzip.

On the server side the aiohttp Server will set Content-Encoding based on mimetype, which for gzip'd files with be 'gzip'.

--web_fileresponse.py:170--
if hdrs.CONTENT_TYPE not in self.headers:
ct, encoding = mimetypes.guess_type(str(filepath))

This will happen regardless of Accept-Encoding in the request header (which might be considered a bug in aiohttp). As you can see in the above code snippet one way to avoid this automatic setting of Content-Encoding is to set the Content-Type in the header, which is what we do here.

With this change in place the Accept-Encoding appears to be respected, presenting the client with the expected outcomes based on the various RFCs. Which ultimately results in a compressed file not being deflated by the aiohttp client. NOTE the Content-Length, Content-Type and Content-Encoding in the following data grabbed from a scenario served by pulp-smash.

--
curl -I -H 'Accept-Encoding: gzip' http://127.0.0.1:$PORT/debian/dists/nosuite/asgard/binary-ppc64/Packages.gz \ && curl -I -H 'Accept-Encoding: identity' http://127.0.0.1:$PORT/debian/dists/nosuite/asgard/binary-ppc64/Packages.gz \ && curl -I -H 'Accept-Encoding: gzip' http://127.0.0.1:$PORT/debian/dists/nosuite/asgard/binary-ppc64/Packages \ && curl -I -H 'Accept-Encoding: identity' http://127.0.0.1:$PORT/debian/dists/nosuite/asgard/binary-ppc64/Packages

HTTP/1.1 200 OK
content-type: application/octet-stream
Etag: "1750f481b8f1ae00-363"
Last-Modified: Wed, 29 Mar 2023 17:38:19 GMT
Content-Length: 867
Accept-Ranges: bytes
Date: Fri, 14 Apr 2023 16:03:49 GMT
Server: Python/3.10 aiohttp/3.8.4

HTTP/1.1 200 OK
content-type: application/octet-stream
Etag: "1750f481b8f1ae00-363"
Last-Modified: Wed, 29 Mar 2023 17:38:19 GMT
Content-Length: 867
Accept-Ranges: bytes
Date: Fri, 14 Apr 2023 16:03:49 GMT
Server: Python/3.10 aiohttp/3.8.4

HTTP/1.1 200 OK
content-type: application/octet-stream
Content-Encoding: gzip
Vary: Accept-Encoding
Etag: "1750f481b8f1ae00-363"
Last-Modified: Wed, 29 Mar 2023 17:38:19 GMT
Content-Length: 867
Accept-Ranges: bytes
Date: Fri, 14 Apr 2023 16:03:49 GMT
Server: Python/3.10 aiohttp/3.8.4

HTTP/1.1 200 OK
content-type: application/octet-stream
Etag: "1750f481b8f1ae00-6d9"
Last-Modified: Wed, 29 Mar 2023 17:38:19 GMT
Content-Length: 1753
Accept-Ranges: bytes
Date: Fri, 14 Apr 2023 16:03:49 GMT
Server: Python/3.10 aiohttp/3.8.4

Note that uncompressed data (Packages) will be compressed during transmission when Accept-Encoding is 'gzip'. This change should not affect tests based off pulp-smash, but if they do break it is more likely the test was working around this issue.

fixes pulp#1309

If a test makes use of a fixture server where the test data/file being
served is already compressed the client will erroneously decompress the
file. The following describes why.

On the client side, the aiohttp Client will automatically decompress
response bodies when the Content-Encoding is one of the compressed
types, like gzip.

On the server side the aiohttp Server will set Content-Encoding based
on mimetype, which for gzip'd files with be 'gzip'.

--web_fileresponse.py:170--
        if hdrs.CONTENT_TYPE not in self.headers:
            ct, encoding = mimetypes.guess_type(str(filepath))
--

This will happen regardless of Accept-Encoding in the request header
(which might be considered a bug in aiohttp). As you can see in the
above code snippet one way to avoid this automatic setting of
Content-Encoding is to set the Content-Type in the header, which is
what we do here.

With this change in place the Accept-Encoding appears to be respected,
presenting the client with the expected outcomes based on the various
RFCs. Which ultimately results in a compressed file not being deflated
by the aiohttp client. NOTE the Content-Length, Content-Type and
Content-Encoding in the following data grabbed from a scenario served
by pulp-smash.

--
curl -I -H 'Accept-Encoding: gzip' http://127.0.0.1:$PORT/debian/dists/nosuite/asgard/binary-ppc64/Packages.gz \
&& curl -I -H 'Accept-Encoding: identity' http://127.0.0.1:$PORT/debian/dists/nosuite/asgard/binary-ppc64/Packages.gz \
&& curl -I -H 'Accept-Encoding: gzip' http://127.0.0.1:$PORT/debian/dists/nosuite/asgard/binary-ppc64/Packages \
&& curl -I -H 'Accept-Encoding: identity' http://127.0.0.1:$PORT/debian/dists/nosuite/asgard/binary-ppc64/Packages

HTTP/1.1 200 OK
content-type: application/octet-stream
Etag: "1750f481b8f1ae00-363"
Last-Modified: Wed, 29 Mar 2023 17:38:19 GMT
Content-Length: 867
Accept-Ranges: bytes
Date: Fri, 14 Apr 2023 16:03:49 GMT
Server: Python/3.10 aiohttp/3.8.4

HTTP/1.1 200 OK
content-type: application/octet-stream
Etag: "1750f481b8f1ae00-363"
Last-Modified: Wed, 29 Mar 2023 17:38:19 GMT
Content-Length: 867
Accept-Ranges: bytes
Date: Fri, 14 Apr 2023 16:03:49 GMT
Server: Python/3.10 aiohttp/3.8.4

HTTP/1.1 200 OK
content-type: application/octet-stream
Content-Encoding: gzip
Vary: Accept-Encoding
Etag: "1750f481b8f1ae00-363"
Last-Modified: Wed, 29 Mar 2023 17:38:19 GMT
Content-Length: 867
Accept-Ranges: bytes
Date: Fri, 14 Apr 2023 16:03:49 GMT
Server: Python/3.10 aiohttp/3.8.4

HTTP/1.1 200 OK
content-type: application/octet-stream
Etag: "1750f481b8f1ae00-6d9"
Last-Modified: Wed, 29 Mar 2023 17:38:19 GMT
Content-Length: 1753
Accept-Ranges: bytes
Date: Fri, 14 Apr 2023 16:03:49 GMT
Server: Python/3.10 aiohttp/3.8.4
--

Note that uncompressed data (Packages) will be compressed during
transmission when Accept-Encoding is 'gzip'. This change should not
affect tests based off pulp-smash, but if they do break it is more
likely the test was working around this issue.

Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com>
@lubosmj lubosmj merged commit a18afe9 into pulp:main Apr 18, 2023
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Respect Accept-Encoding for already compressed content
2 participants