-
-
Notifications
You must be signed in to change notification settings - Fork 36
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Poor performance on large monorepos #115
Comments
@squidfunk I know you did this in several of your plugins. Any wisdom to share? |
In Material for MkDocs, the privacy plugin, optimize plugin and new social plugin make heavy use of concurrent futures and caching of (partial) results. It's a new technique I learned when first writing the optimize plugin. The general idea is to split off work into threads where possible, and only reconcile jobs when necessary. Examples:
I plan to write a blog post about my learnings in writing MkDocs plugins in the future. |
I have a prototype patch for the time stamp one where I do something similar. I'm using on_files to compute time stamps for all files, finding that 10 is about the max parallelism that works reliably. As a test case a site with 78 markdown files in a large monorepo takes 5 seconds to generate with backstage techdocs cli. With the time stamp plug-in this increased to 378 seconds. With the parallel precomputed time stamp patch this goes down to 69 seconds -- still quite a big hit but a big improvement. |
When trying to use this on a large monorepo, performance is very poor -- one docs site with ~30 pages has gone from taking a a few seconds to build to a few minutes.
I'm testing a change that would parallelize calls to git log up front once we have the list of files
on_files
since mkdocs itself can't be parallelized to runon_page_markdown
in a multi-threaded fashion.Curious if anyone else has run into this and if this is something that would be a useful contribution via PR.
The text was updated successfully, but these errors were encountered: