bpo-43014: Improve performance of tokenize by 20-30%

[asottile]

https://bugs.python.org/issue43014

[isidentical]

Great optimization, though there are 2 concerns of mine;

• For people who are not using tokenize module to generate tokens (like detect_encoding/open are the most common functions), they'd have to pay this cost
• Also, even though breaking them is somewhat OK, there are wild usages out there that monkeypatches the PseduoToken to change the behavior (add new tokens) of tokenize module.

Maybe there is a solution that would both optimize this, and also don't cause any new regressions for normal users (something like @lru_cache to _compile maybe?)

[asottile]

I initially approached this with lru_cache, however the function call alone accounts for 6% of the execution so the performance gains aren't as significant

[isidentical]

I initially approached this with lru_cache, however the function call alone accounts for 6% of the execution so the performance gains aren't as significant

Maybe we could set it to a global (_PSEDUO_TOKEN_RE = None, if ... is None: _PSEDUO_TOKEN_RE = compile())?

[asottile]

I initially approached this with lru_cache, however the function call alone accounts for 6% of the execution so the performance gains aren't as significant

Maybe we could set it to a global (_PSEDUO_TOKEN_RE = None, if ... is None: _PSEDUO_TOKEN_RE = compile())?

from my tests this performs the same as the lru_cache approach (within a few 1s of ms -- error noise). the lru_cache approach seems a reasonable middle ground (and also avoids recompiling the triple-quoted-string regexes over and over as well)

[isidentical]

Thanks a lot @asottile!

[pablogsal]

I think we should limit explicitly (I am aware there is a default of 128) the max size of the cache in this call so this doesn't blow up for some super intensive cases.

[pablogsal]

In the re module we use the same trick and we have 512 as the max size of the cache.

[asottile]

I commented on the other PR, there's only 5 possible regex strings that go through this call

[pablogsal]

Roger. Then is fine, although I would have preferred to manually pre-compile those and avoid importing functools

[asottile]

that was my original patch, it increased import time of this module by ~20ms

[pablogsal]

How much is importing functools? Also, we can still go that way if we do the compilation lazily, but I don't think is worth it.

[asottile]

it is already imported via tokenize => re => functools so 0

[pablogsal]

Oh, then I retract my concern. This is an absolutely better approach. Thanks for following with me :)