Brotli encoding is not recognized by mimetypes

[Andrey]

BPO	32021
Nosy	@tiran, @bitdancer, @maxking, @csabella
PRs	bpo-32021: Support brotli .br encoding in mimetypes #12200

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-01-12.22:33:29.295>
created_at = <Date 2017-11-14.07:43:50.631>
labels = ['type-feature', 'library', '3.9']
title = 'Brotli encoding is not recognized by mimetypes'
updated_at = <Date 2020-01-12.22:33:29.293>
user = 'https://bugs.python.org/Andrey'

bugs.python.org fields:

activity = <Date 2020-01-12.22:33:29.293>
actor = 'maxking'
assignee = 'none'
closed = True
closed_date = <Date 2020-01-12.22:33:29.295>
closer = 'maxking'
components = ['Library (Lib)']
creation = <Date 2017-11-14.07:43:50.631>
creator = 'Andrey'
dependencies = []
files = []
hgrepos = []
issue_num = 32021
keywords = ['patch']
message_count = 13.0
messages = ['306188', '306189', '306195', '306203', '306215', '306221', '306222', '306226', '306239', '306251', '306293', '359766', '359880']
nosy_count = 5.0
nosy_names = ['christian.heimes', 'r.david.murray', 'maxking', 'cheryl.sabella', 'Andrey']
pr_nums = ['12200']
priority = 'normal'
resolution = None
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue32021'
versions = ['Python 3.9']

[Andrey]

Brotli (.br) encoding is not recognized by mimetypes module.
mimetypes doesn't have API for adding encodings.
The encoding is supported by most browsers: https://caniuse.com/#feat=brotli

[tiran]

You can just add the encoding to mimetypes.encodings_map:

mimetypes.encodings_map['.br'] = 'brotli'

[Andrey]

Yes, however would it make sense to add it, so mimetypes recognizes it by default? I will make a PR if it makes sense.

[bitdancer]

It looks like Brotli is a de-facto standard (RFC 7932 is an informational RFC), and it says the IANA registry has been updated. However, this appears to be http-only, with no corresponding program used for compression of files, and thus no "file extension" to be registered, or program name to put in the encodings_map table. We don't have most of the types listed in the http content encodings registry in mimetypes, so I don't at this time see any reason to add br.

mimetypes is aimed at files, not http. In what context do you want to use mimetimes to look up br?

[Andrey]

When "type, encoding = mimetypes.guess_type('file.js.br')" is used not only the "encoding" is not set to "brotli", but the type is also not recognized.
Such code is used in aiohttp for example.

My proposal is to add 'br' to mimetypes.encodings_map, not to mimetypes.types_map

[bitdancer]

OK, so there are "real" filenames that have .br on the end? That wasn't clear from the RFC and discussions I found. Based on the documentation, br is for use in the Content-Encoding header.

What does aiohttp do when it gets a brotli encoded file? What does it do with the other types in the http content-encoding table? (https://www.iana.org/assignments/http-parameters/http-parameters.xhtml#content-coding)

Where is the standard that talks about the '.br' file extension?

[bitdancer]

Note, I'm not saying we shouldn't do this, I'd just like to get good documentation of why linked to this issue.

[Andrey]

Currently aiohttp doesn't support Brotli. (I'm here because I would like to get that support)

When it gets (as a server) a request for file.html it checks if file.html.gz exists (and accept-encoding contains 'gzip'). It then send file.html.gz to the client and sets Content-Encoding to gzip.

It uses mimetypes.guess_type to set Content-Type and Content-Encoding.

[bitdancer]

Hmm. OK, the docs do say that value should be usable as a content-encoding header.

However, absent any sort of program that actually does brotli encoding/decoding to and from files, this would be essentially useless based on the aiohttp's algorithm (or any other use I can think of for the mimetimes module in this context). Does such a program exist? I didn't find one searching the web.

[Andrey]

NodeJS: https://www.npmjs.com/package/brotli
Python:
https://pypi.python.org/pypi/brotlipy
https://pypi.python.org/pypi/Brotli
Ubuntu: https://launchpad.net/ubuntu/+source/brotli

That said I don't expect an average user to compress their files as .br, but it is useful for site/webapp developers to have static resources as .br

[bitdancer]

OK, bouncing around from those URL I finally found this:

google/ngx_brotli#28

That indicates unambiguously that someone does expect files to be pre-compressed on the server and that the main brotli package provides a cli tool (I could not find any reference to that in the brotli docs themselves, such as they are).

So, I think it is appropriate to add this, and one can hope that the program name is indeed 'brotli' so that it matches up with the existing mimetypes docs :)

[csabella]

Now that two years have passed, is there any additional info as to whether this should be added or not? It does seem that the format is active (https://github.com/google/brotli).

[maxking]

New changeset b2b4a51 by Abhilash Raj (Philip McMahon) in branch 'master':
bpo-32021: Support brotli .br encoding in mimetypes (bpo-12200)
b2b4a51