New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Building unchanged documentation takes very long #14204
Comments
comment:1
I don't know. I think there are 74 documents that need to be built: 56 modules in the reference manual, 1 master reference manual doc, and 17 others. When you run I wonder if it could be sped up much by calling Sphinx internally in Python, instead of as a command-line call each time. Otherwise, I don't have any ideas. Users should be advised to run Should this really be a blocker? |
comment:2
For what it's worth, you can cut down on the build time by setting
and
|
comment:3
Replying to @jhpalmieri:
That's a personal opinion of course. In my opinion, the answer is absolutely "Yes". If you regularly run |
comment:4
How long does the rest of |
comment:5
Replying to @jhpalmieri:
OK, with 4 processes on
It's not that those extra minutes matter that much, but those lost minutes matter much more to me than the minutes saved by #6495. |
comment:6
Replying to @jhpalmieri:
Do you have any idea what Sphinx does in those 6 seconds? Does it import something from |
comment:7
If I put a call to
If I put it instead into the Python command called in builder.py, I get more information:
We're calling certain functions a huge number of times, but we're not importing sage.all (as far as I can tell). I don't know why we're calling posixpath.join 35000 times, for example: that seems like a lot. Could it be reading every file in the Sage library each time, instead of only the files referenced in that section? |
comment:8
Actually, it is importing sage.all every time. |
comment:9
I think the issue is
Could we replace this subprocess stuff by a simple Python function call or an |
comment:10
Replying to @jdemeyer:
I tried to do that, but I think Sphinx has a memory leak (this has been suggested before). For whatever reason, after a little while it uses a lot of memory and slows way down. Actually, the inventory build sped up (from almost 2 minutes before down to about half a minute on an OS X machine with 2 cores), but the html build does not. In more detail, I tried importing sage.all once, and then using a Python function call to build each document: I basically stuck all of custom-sphinx-builder.py into builder.py, and replaced the
This way sage.all would be imported just once, not 75 times. But there are issues. Maybe we should call |
comment:11
Replying to @jhpalmieri:
Use |
comment:12
My laptop builds the new-style documentation in 55 seconds with 8x parallel if nothing is changed. And this is from cold caches. Jeroen, you are just trying to replace the subprocess call with a fork. That just saves us a couple of Sage startups, each of which is 1.5 seconds. Thats maybe 10% performance that you can get out of there, right? You need to pull in a number of files across the different doc pieces to figure out if stuff is out-of-date, so I guess this makes this ticket extremely dependent on small file read performance. Thats just the price of having to synchronize multiple separate sub-documents. William, is there a chance to get a SSD (presumably into sage.math) for Jeroen? |
comment:13
Replying to @vbraun:
Which is probably about the same time as one process for the old docbuilder.
Sure.
I doubt it. My timings suggest that some documents need 2 ~ 3 seconds to build, which is pretty close to the Sage startup time. So I think there is a substantial gain there. I think the elephant in the room is still: why did we care so much about speeding up the initial build of the documentation (#6495) at the expense of rebuilds of the documentation? When doing a few rebuilds of the documentation, you lose all the time which is gained by #6495. |
comment:14
When I was thinking about #6495, I was imagining the following situations, among others:
For all of these, #6495 helps considerably. When I think about running |
comment:15
Could we at the very least get rid of the |
comment:16
Replying to @vbraun:
There already is a SSD in |
comment:17
This is a proof-of-concept patch for getting rid of |
comment:18
Perhaps more imports could be done it the master process, but I don't know how much there is to gain. Running |
comment:19
Add a cache for the intersphinx inventory, reducing the total time again by more than a factor of 2. Still untested proof-of-concept. |
comment:20
Patch seems to work, I'm setting it to needs_review very tentatively. Somebody who knows Sphinx better than me should look at it. With this patch, building the documentation with 2 threads is now faster than before #6495, so it would fix the issue. |
Author: Jeroen Demeyer |
comment:21
The patch seems to work as advertised, and the changes make sense to me. Volker, do you concur? (Note that the commit message is not ideal ;) |
Attachment: 14204_sphinx_fork.patch.gz |
comment:22
Fixed commit message. |
Reviewer: John Palmieri |
Merged: sage-5.8.beta4 |
comment:25
I have a followup question. I'm working with a student on learning to develop, and his first patch is a pretty basic documentation upgrade. Changing just one file and then doing |
comment:26
All the more reason to set
With
On sage.math, with
So using multiple cores is very advantageous. Note also that if you are only modifying a single file, or files within one section of the reference manual, be aware that you can do
(or |
comment:27
Replying to @kcrisman:
On my reasonably modern Gentoo laptop, this takes about 40 seconds (with 1 thread), which I personally find reasonable.
I don't think it's such a big regression. With 1 thread, it is slightly slower than before, but not by much. If you do want to create a ticket, this is the place to do it: http://bitbucket.org/birkenfeld/sphinx/issues/ |
comment:28
I see, so this isn't something we can fix more without getting rid of parallel documentation building. I think this was a lot faster before because it only had to do a sweep through one directory. With Sage 5.2, after touching one file and Maybe we could add something about both of John's solutions to the Sage manuals developer page? I'd be open to opening such a ticket. Maybe my student could even write the patch ;-) |
Even with #14156, rebuilding the documentation when nothing needs to be done (
make doc
right aftermake doc
) still takes a very long time.In sage-5.7:
In sage-5.8.beta2:
CC: @jhpalmieri @vbraun @hivert @williamstein
Component: build
Author: Jeroen Demeyer
Reviewer: John Palmieri
Merged: sage-5.8.beta4
Issue created by migration from https://trac.sagemath.org/ticket/14204
The text was updated successfully, but these errors were encountered: