Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove filename pattern caches #2153

Merged
merged 1 commit into from
Jun 4, 2022
Merged

Conversation

jeanas
Copy link
Contributor

@jeanas jeanas commented May 29, 2022

These are irrelevant because the fnmatch module implementation already
uses functools.lru_cache.

(More diff size reduction towards a new plugin system.)

These are irrelevant because the fnmatch module implementation already
uses functools.lru_cache.
Copy link
Member

@birkenfeld birkenfeld left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

@birkenfeld birkenfeld merged commit 951c894 into pygments:master Jun 4, 2022
@Anteru Anteru added this to the 2.13.0 milestone Jun 19, 2022
dirkmueller added a commit to dirkmueller/pygments that referenced this pull request Jan 30, 2023
d0487e3 "Remove filename pattern caches" (pygments#2153)
introduced a huge performance regession. While it is true
that fnmatch already uses functools.lru_cache, that cache
is limited to 127 entries and we have over 1000 matching patterns,
which means the cache is evicted entirely on every iteration.

We can be more clever without reverting that patch by avoiding
fnmatch calls alltogether, which provides an even better speedup.
The bulk (> 99%) of all filename matches are on the filename extension,
so in gen_mapfiles.py we split these into a separate tuple list that
we match without calling fnmatch().

This restores previous performance speedup without the overhead
of an extra regexp cache.
dirkmueller added a commit to dirkmueller/pygments that referenced this pull request Jan 31, 2023
d0487e3 "Remove filename pattern caches" (pygments#2153)
introduced a huge performance regession. While it is true
that fnmatch already uses functools.lru_cache, that cache
is limited to 127 entries and we have over 1000 matching patterns,
which means the cache is evicted entirely on every iteration.

We can be more clever without reverting that patch by avoiding
fnmatch calls alltogether, which provides an even better speedup.
The bulk (> 99%) of all filename matches are on the filename extension,
so in gen_mapfiles.py we split these into a separate tuple list that
we match without calling fnmatch().

This restores previous performance speedup without the overhead
of an extra regexp cache.
dirkmueller added a commit to dirkmueller/pygments that referenced this pull request Jan 31, 2023
This introduced a performance regession. While it is true that
fnmatch already uses functools.lru_cache, that cache is limited to 256
on python 3.10 and older and we have over 1000 matching patterns, which
means the cache is evicted entirely on every iteration.

This reverts commit 951c894.
jeanas pushed a commit that referenced this pull request Jan 31, 2023
This introduced a performance regession. While it is true that
fnmatch already uses functools.lru_cache, that cache is limited to 256
on python 3.10 and older and we have over 1000 matching patterns, which
means the cache is evicted entirely on every iteration.

This reverts commit 951c894.
@jeanas jeanas deleted the fncache branch September 1, 2023 16:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants