Skip to content

Commit

Permalink
pulp3: fixture_utils.py: avoid automatic encoding
Browse files Browse the repository at this point in the history
fixes #1309

If a test makes use of a fixture server where the test data/file being
served is already compressed the client will erroneously decompress the
file. The following describes why.

On the client side, the aiohttp Client will automatically decompress
response bodies when the Content-Encoding is one of the compressed
types, like gzip.

On the server side the aiohttp Server will set Content-Encoding based
on mimetype, which for gzip'd files with be 'gzip'.

--web_fileresponse.py:170--
        if hdrs.CONTENT_TYPE not in self.headers:
            ct, encoding = mimetypes.guess_type(str(filepath))
--

This will happen regardless of Accept-Encoding in the request header
(which might be considered a bug in aiohttp). As you can see in the
above code snippet one way to avoid this automatic setting of
Content-Encoding is to set the Content-Type in the header, which is
what we do here.

With this change in place the Accept-Encoding appears to be respected,
presenting the client with the expected outcomes based on the various
RFCs. Which ultimately results in a compressed file not being deflated
by the aiohttp client. NOTE the Content-Length, Content-Type and
Content-Encoding in the following data grabbed from a scenario served
by pulp-smash.

--
curl -I -H 'Accept-Encoding: gzip' http://127.0.0.1:$PORT/debian/dists/nosuite/asgard/binary-ppc64/Packages.gz \
&& curl -I -H 'Accept-Encoding: identity' http://127.0.0.1:$PORT/debian/dists/nosuite/asgard/binary-ppc64/Packages.gz \
&& curl -I -H 'Accept-Encoding: gzip' http://127.0.0.1:$PORT/debian/dists/nosuite/asgard/binary-ppc64/Packages \
&& curl -I -H 'Accept-Encoding: identity' http://127.0.0.1:$PORT/debian/dists/nosuite/asgard/binary-ppc64/Packages

HTTP/1.1 200 OK
content-type: application/octet-stream
Etag: "1750f481b8f1ae00-363"
Last-Modified: Wed, 29 Mar 2023 17:38:19 GMT
Content-Length: 867
Accept-Ranges: bytes
Date: Fri, 14 Apr 2023 16:03:49 GMT
Server: Python/3.10 aiohttp/3.8.4

HTTP/1.1 200 OK
content-type: application/octet-stream
Etag: "1750f481b8f1ae00-363"
Last-Modified: Wed, 29 Mar 2023 17:38:19 GMT
Content-Length: 867
Accept-Ranges: bytes
Date: Fri, 14 Apr 2023 16:03:49 GMT
Server: Python/3.10 aiohttp/3.8.4

HTTP/1.1 200 OK
content-type: application/octet-stream
Content-Encoding: gzip
Vary: Accept-Encoding
Etag: "1750f481b8f1ae00-363"
Last-Modified: Wed, 29 Mar 2023 17:38:19 GMT
Content-Length: 867
Accept-Ranges: bytes
Date: Fri, 14 Apr 2023 16:03:49 GMT
Server: Python/3.10 aiohttp/3.8.4

HTTP/1.1 200 OK
content-type: application/octet-stream
Etag: "1750f481b8f1ae00-6d9"
Last-Modified: Wed, 29 Mar 2023 17:38:19 GMT
Content-Length: 1753
Accept-Ranges: bytes
Date: Fri, 14 Apr 2023 16:03:49 GMT
Server: Python/3.10 aiohttp/3.8.4
--

Note that uncompressed data (Packages) will be compressed during
transmission when Accept-Encoding is 'gzip'. This change should not
affect tests based off pulp-smash, but if they do break it is more
likely the test was working around this issue.

Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com>
  • Loading branch information
masselstine authored and lubosmj committed Apr 18, 2023
1 parent e1ed6c0 commit a18afe9
Showing 1 changed file with 4 additions and 1 deletion.
5 changes: 4 additions & 1 deletion pulp_smash/pulp3/fixture_utils.py
@@ -1,4 +1,5 @@
from aiohttp import web
from multidict import CIMultiDict


def add_file_system_route(app, fixtures_root):
Expand All @@ -13,7 +14,9 @@ async def all_requests_handler(request):
requests.append(request)
path = fixtures_root / request.raw_path[1:] # Strip off leading '/'
if path.is_file():
return web.FileResponse(path)
return web.FileResponse(
path, headers=CIMultiDict({"content-type": "application/octet-stream"})
)
else:
raise web.HTTPNotFound()

Expand Down

0 comments on commit a18afe9

Please sign in to comment.