Remove case sensitivity of Etag header for use_etag

[nicholasmhughes]

What does this PR do?

See issue.

What issues does this PR fix or reference?

Fixes: #61440

Merge requirements satisfied?

[NOTICE] Bug fixes or features added to Salt require tests.

• Docs
• Changelog - https://docs.saltproject.io/en/master/topics/development/changelog.html
• Tests written/updated

Commits signed with GPG?

Yes

Please review Salt's Contributing Guide for best practices.

See GitHub's page on GPG signing for more information about signing commits with GPG.

[nicholasmhughes]

re-run full all

[waynew]

👍

[waynew]

re-run all

[damon-atkins]

Hi @nicholasmhughes thanks for doing this. A better method would be to write the whole header to a different file instead of using a file specific for etag. There are many ways to determine if a file should be cached from a web server. Keeping the whole header in a filename.meta file allows not only etag to be used but other techniques, like if-not-modified-since using the date from save header which is in the same format as what the web server provided (instead of try to create it). Instead of use_etag just have http_cache. e.g.

Here is some sample code, loading meta data, back into a case insensitive data structure.

meta_file = '{0}.meta'.format(dest_file)
with salt.utils.files.fopen(meta_file, 'r', encoding='utf-8') as meta_fh:
   cache_headers = requests.structures.CaseInsensitiveDict(data=salt.utils.json.load(meta_fh))

Not sure why salt does not use the request library.

How are you handling failures/partial downloads?

[nicholasmhughes]

@damon-atkins , totally understand the broader metadata perspective. However, I'm not really in a spot to address that at this time.

Regarding handling failures/partial downloads, the code additions for use_etag aren't in the critical path for content download handling itself... just the decision on whether or not to download. If an etag file isn't there, is corrupted, or otherwise "partial", then it'll be treated as not matching and re-download the content.