Make SimpleHTTPRequestHandler load mimetypes lazily

[zooba]

BPO	35292
Nosy	@pfmoore, @tjguk, @pmp-p, @zware, @zooba, @demianbrecht, @danishprakash
PRs	bpo-35292: Lazily inject system mime.types into SimpleHTTPRequestHandler.extensions_map #11036 bpo-35292: Avoid call mimetypes.init when http.server is imported #17822

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-01-08.18:31:52.576>
created_at = <Date 2018-11-21.17:41:40.354>
labels = ['easy', 'library', '3.9', 'OS-windows', 'performance']
title = 'Make SimpleHTTPRequestHandler load mimetypes lazily'
updated_at = <Date 2020-01-08.18:31:52.575>
user = 'https://github.com/zooba'

bugs.python.org fields:

activity = <Date 2020-01-08.18:31:52.575>
actor = 'steve.dower'
assignee = 'none'
closed = True
closed_date = <Date 2020-01-08.18:31:52.576>
closer = 'steve.dower'
components = ['Library (Lib)', 'Windows']
creation = <Date 2018-11-21.17:41:40.354>
creator = 'steve.dower'
dependencies = []
files = []
hgrepos = []
issue_num = 35292
keywords = ['patch', 'easy']
message_count = 15.0
messages = ['330212', '330292', '330322', '331381', '331386', '331392', '331394', '331395', '331396', '331398', '331399', '331400', '331739', '359621', '359622']
nosy_count = 7.0
nosy_names = ['paul.moore', 'tim.golden', 'pmpp', 'zach.ware', 'steve.dower', 'demian.brecht', 'danishprakash']
pr_nums = ['11036', '17822']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue35292'
versions = ['Python 3.9']

[zooba]

Importing http.server triggers mimetypes.init(), which can be fairly expensive on Windows (and potentially other platforms?) due to having to enumerate a lot of registry keys.

Instead, SimpleHTTPRequestHandler.extensions_map can be a dict with just its default values in lib/http/server.py and the first call to guess_type() can initialize mimetypes if necessary.

We should check whether people read from SimpleHTTPRequestHandler.extensions_map directly before calling guess_type(), and decide how quickly we can make the change. My gut feeling is we will be okay to make this in the next release but probably shouldn't backport it.

[danishprakash]

I would like to work on this if you have not already started, Steve.

We should check whether people read from SimpleHTTPRequestHandler.extensions_map directly before calling guess_type(), and decide how quickly we can make the change.

Are you implying we make the change based on whether someone use SimpleHTTPRequestHandler.extensions_map before calling guess_type()? Could you please elaborate that a bit, thanks.

[zooba]

You're welcome to it - I deliberately left it for someone else to work on, though I'm happy to review and merge.

The visible change of making this lazy is that if someone reads from the dict, it'll be missing system-specified content types (until we lazily fill it in guess_type() ). Nobody is meant to read from it - it's public for people to *add* custom overrides - so this should be okay, but a little bit of GitHub research to find out if anyone does wouldn't hurt. Then we can at least warn them that it's changing before they run into their own bug.

[pmp-p]

and potentially other platforms?
strangely, it does not.

but adding a blocking read on first requests may ruin profiling data on any platform.

isn't http.server reserved for instrumentation and testing ?

[zooba]

Instrumentation is not the same thing as profiling, and this one is certainly not intended for performance. That said, simply importing it shouldn't impact anyone else's performance either, and since six imports this, fixing it will have a pretty broad impact.

Thanks for the PR - I'll look when I get a chance, but most of my time is going towards the next 3.7 release right now. Feel free to ping me on here in a week if nobody has had a look.

[pmp-p]

??? i never compared Instrumentation and profiling

and what getting delta timing about the script behind the first http request has to do with performance ? <= that's actually another question

thefirst was : isn't http.server reserved for instrumentation and testing (because lazy loading seems a production feature to me ) ?

[zooba]

If you weren't conflating profiling and instrumentation then your original comment makes no sense.

[zooba]

(Sorry, page submitted while I was still typing for some reason.)

... Instrumentation for everything other than time is unaffected, so if you didn't mean to conflate them, then no there is no impact on the documented use.

Yes, moving mimetypes.init to the first request will make the first request slower. However, it makes *import* faster, and there should not be a significant penalty just for importing this module.

You can trivially front-load the work in your app by calling mimetypes.init yourself, which will reduce the overhead on first request to a dict copy (I assume, haven't looked at the PR yet). Or just don't measure the first request - it's typical to do some warm-up loops when profiling.

[pmp-p]

the PR as it is actually add a blocking read (and maybe exceptions or whatever mimetypes modules decide to do) in processing of the first request to server, which has nothing to do with the code serving that request.

I agree that makes no sense at all.

[zooba]

I agree that makes no sense at all.

I have to admit I have no idea what you're trying to argue or suggesting as an alternative. Could you try clearly explaining the change you'd rather see to fix the issue?