Fix mirror=True not including files declared in .treeinfo in publication

[dralley]

closes: #9230
https://pulp.plan.io/issues/9230

[dralley]

CI likely to fail for same reason as here: #2087 (comment)

[dralley]

Epic fail, this wasn't actually testing the mirrored publication :( I was doing testing manually also so nothing was missed, and the test wouldn't have caught this, but still.

[pulpbot]

Attached issue: https://pulp.plan.io/issues/9230

[dralley]

According to https://github.com/pulp/pulp_rpm/blob/2-master/plugins/pulp_rpm/plugins/importers/yum/parse/treeinfo.py#L542-L565, we may need to also handle the possibility of files not present in checksums but declared elsewhere.

We're also missing discinfo and EFI/* because neither they nor their contents are declared in any metadata files. Do we need to be concerned about those? What are they used for? (They 100% aren't part of any normal publishes so if we haven't heard anything yet we're probably OK)

Lastly I noticed that if I actually download the file with http I get the following

(pulp) (pulp) [vagrant@pulp3-source-fedora34 devel]$ http GET :24816/pulp/content/test/images/pxeboot/vmlinuz
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Encoding: gzip
Content-Type: text/plain
Date: Sun, 22 Aug 2021 03:15:11 GMT
EJ-HOST: authorizer-prod-dc-us-east-20-c27lh
Etag: "3b95cceb4a3d43e1ce9cd7beda8e3f4f:1620737008.051679"
Last-Modified: Thu, 29 Apr 2021 13:03:45 GMT
Server: AkamaiNetStorage
Transfer-Encoding: chunked
Vary: Accept-Encoding
X-Akamai-Request-ID: e01bfe4
X-Cache: TCP_HIT from a23-43-58-101.deploy.akamaitechnologies.com (AkamaiGHost/10.4.4-34529956) (-)


http: error: ContentDecodingError: ('Received response with content-encoding: gzip, but failed to decode it.', error('Error -3 while decompressing data: incorrect header check'))

If I download from Pulp with wget then it works - and then afterwards downloading with httpie works fine. I suspect I may have figured out the cause of https://pulp.plan.io/issues/9213 - perhaps we forward the headers when streaming the content, including TransferEncoding, but we're decompressing in the middle. Which would explain why it was trying to un-gzip data that isn't gzipped, and might explain the size discrepancy.

[goosemania]

Thanks for the fix!