Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize getting relative page URLs, now with less custom code #2407

Merged
merged 2 commits into from Jul 7, 2021

Conversation

oprypin
Copy link
Member

@oprypin oprypin commented May 16, 2021

This is a custom implementation that's significantly faster but always gives the exact same results as the current one. The use of posixpath.relpath pulled in several path-specific transformations that are never needed here.

Efficiency is important because calls to normalize_url (e.g.) on a site with ~300 pages currently take up ~10% of the total run time due to the sheer number of them. The number of calls is at least the number of pages squared.

Generally, approximating that the number of these calls is N×N×2, this is the graph that we end up with:
image
Full source code how I got this result

This shows the total time spent getting relative paths (Y axis) over the course of building a site with that many pages (X axis).
With red color being "before" and green color being "after".
This does not show the total site build times, rather you can only subtract "red" from "green" to approximate the absolute savings of time.

In all other regards the sites' build times grow linearly, but only this particular place it grows as N^2 (because the templates end up linking to every page on every built page). So, with more and more pages, more and more percentage of time is spent just generating these relative URLs.
So I'm making it very optimal, because it's the place that really matters.

Previously: #2272, #2296 (a big number of tests was already added there)

This is a custom implementation that's significantly faster but gives the exact same results as the current one. The use of `posixpath.relpath` pulled in several path-specific transformations that are never needed here.

Efficiency is important because calls to `normalize_url` (e.g.) on a site with ~300 pages currently take up ~10% of the total run time due to the sheer number of them. The number of calls is at least the number of pages squared.
@oprypin
Copy link
Member Author

oprypin commented Jul 7, 2021

I intend to merge this soon

@oprypin oprypin merged commit f6d8830 into mkdocs:master Jul 7, 2021
@oprypin oprypin deleted the relpath branch July 7, 2021 16:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant