[utils] Don't trust Content-length when Content-encoding is present

[felixonmars]

IMPORTANT: PRs without the template will be CLOSED

Description of your pull request and other information

Our YoutubeDLHandler is doing auto-decompression, so that we cannot compare Content-length to the size of extracted data. Although HttpFD has specified Youtubedl-no-compression, the web server may still ignore this and return deflated data (as the situation in #3772).

This change attempts to retain the Content-encoding header for later comparison in HttpFD so that we could discard data_len when the web server returns compressed data. I didn't find the reason why this header was previously removed but didn't yet find a counter example either.

Fixes #3772 in my experiments.

Template

Before submitting a pull request make sure you have:

• At least skimmed through contributing guidelines including yt-dlp coding conventions
• Searched the bugtracker for similar pull requests
• Checked the code with flake8 and ran relevant tests

In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under Unlicense. Check one of the following options:

• I am the original author of this code and I am willing to release it under Unlicense
• I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

• Fix or improvement to an extractor (Make sure to add/update tests)
• New extractor (Piracy websites will not be accepted)
• Core bug fix/improvement
• New feature (It is strongly recommended to open an issue first)

[pukkandan]

cc @coletdjnz

[coletdjnz]

note for self: check against urllib3/requests too

[dirkf]

None is default?

Suggested change

	if ctx.data.info().get('Content-encoding', None):
	if ctx.data.info().get('Content-encoding'):

Or maybe

Suggested change

	if ctx.data.info().get('Content-encoding', None):
	if ctx.data.info().get('Content-encoding') is not None:

[felixonmars]

Yes, I'm just trying to keep the same coding style as data_len is also adding this , None. I'll try to remove both then.

[dirkf]

... I didn't find the reason why this header was previously removed but didn't yet find a counter example either.

When I worked on this code I inferred that it was trying to handle nested encodings, or perhaps more realistically to avoid decoding an already decoded contents. The old yt-dl code was like this:

old_resp = resp
if resp.headers.get('Content-encoding', '') == method1:
    # expand resp using method1
    # set resp.msg to old_resp.msg
    del resp.headers['Content-encoding']
if resp.headers.get('Content-encoding', '') == method2:
    # expand resp using method2
    # set resp.msg to old_resp.msg
    del resp.headers['Content-encoding']
if resp.headers.get('Content-encoding', '') == method3:
    # expand resp using method3
    # set resp.msg to old_resp.msg
    del resp.headers['Content-encoding']
...

[felixonmars]

When I worked on this code I inferred that it was trying to handle nested encodings, or perhaps more realistically to avoid decoding an already decoded contents.

I see. An alternative approach would be moving the info into a custom header like "Youtubedl-decompressed" and check that in HttpFD instead.

[coletdjnz]

urllib3/requests appear to retain the content-encoding header after decompression too, so this should be fine in that regard (needed to check due to #2861 and #3668).

When I worked on this code I inferred that it was trying to handle nested encodings, or perhaps more realistically to avoid decoding an already decoded contents.

Shouldn't be an issue as the data should only be tried to be decoded once (i.e. content-encoding read once for whole decoding process, including multi decoding). I don't see us needing anyway of checking if the content is decoded or not either (and in the case it is in an unsupported encoding, we can catch that elsewhere), so including the header even when decoded should be fine.

[coletdjnz]

Networking changes look fine to me.

#3772 seems to work for me too now assuming you don't use --test (@pukkandan not sure if that is a concern)

[pukkandan]

assuming you don't use --test (@pukkandan not sure if that is a concern)

Let's not worry about that for now

Make sure to port this to the networking PR too