Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Memory hog for Linux documentation with -j1 #11124

Closed
marxin opened this issue Jan 12, 2023 · 5 comments
Closed

Memory hog for Linux documentation with -j1 #11124

marxin opened this issue Jan 12, 2023 · 5 comments

Comments

@marxin
Copy link
Contributor

marxin commented Jan 12, 2023

Describe the bug

As shown here:
https://lore.kernel.org/all/bdd04f2e-64a4-e3ef-eb4c-3681c6f52c2f@gmail.com/

 ======  ===================================  ===============================
                     elapsed time                       maxresident
         -----------------------------------  -------------------------------
 Sphinx     -j1     -j2      -j4      -j6       -j1     -j2     -j4     -j6
 ======  ======== ======== ======== ========  ======= ======= ======= =======
  6.1.3  15:03.83 11:31.99  9:35.15  8:49.01  2949056 1059516  978232  967400
  6.1.2  15:11.74 18:06.89 16:39.93      OOM  2961524 5548344 5255372      --
  5.3.0  14:13.04 10:16.81  8:22.37  8:09.74   711532  937660  846016  800340
 ======  ===================================  ===============================

The Linux documentation build takes much more memory with -j1 compared to 5.3.0 release.

I bisected that to 463a696 (one needs cherry-pick dcb4429)

How to Reproduce

make SPHINXOPTS="-q -j1" htmldocs

Environment Information

Sphinx `6.1.3`

Sphinx extensions

No response

Additional context

No response
@marxin
Copy link
Contributor Author

marxin commented Jan 12, 2023

@AA-Turner

@marxin
Copy link
Contributor Author

marxin commented Apr 4, 2023

I'm attaching CPU and memory graphs for linux documentation (with -j1) with the current caching:

linux-sphinx-cache

and if I reverted the mentioned patch:

linux-sphinx-no-cache

So overall, the caching scheme speeds up the documentation build by 8%.

However, I noticed that with -j auto (j > 2), there is no cache hit for _write_doc_doctree_cache that is likely caused by the usage of a process pool where there is not overlap in between read and write phase.

marxin added a commit to marxin/sphinx that referenced this issue Apr 5, 2023
@marxin
Use _write_doc_doctree_cache only in serial mode.
cd639af 
As shown in sphinx-doc#11124, if the reading operation is using multiple
processes, then the content of self.env._write_doc_doctree_cache
is not synchronized with the main process. Thus use it only
in serial mode.

Related: sphinx-doc#11124
marxin added a commit to marxin/sphinx that referenced this issue Apr 5, 2023
@marxin
Use _write_doc_doctree_cache only in serial mode.
0d085d8 
As shown in sphinx-doc#11124, if the reading operation is using multiple
processes, then the content of self.env._write_doc_doctree_cache
is not synchronized with the main process. Thus use it only
in serial mode.

For cpython documentation I get to the following numbers
before this revision:
-j1 + caching of _write_doc_doctree_cache: 95 s
-j1 + not caching of _write_doc_doctree_cache: 86 s
-j32 + caching of _write_doc_doctree_cache: 57 s
-j32 + no caching of _write_doc_doctree_cache: 57 s

Related: sphinx-doc#11124
marxin added a commit to marxin/sphinx that referenced this issue Apr 5, 2023
@marxin
Use _write_doc_doctree_cache only in serial mode.
936b8c1 
As shown in sphinx-doc#11124, if the reading operation is using multiple
processes, then the content of self.env._write_doc_doctree_cache
is not synchronized with the main process. Thus use it only
in serial mode.

For cpython documentation I get to the following numbers
before this revision:
-j1 + caching of _write_doc_doctree_cache: 95 s
-j1 + not caching of _write_doc_doctree_cache: 86 s
-j32 + caching of _write_doc_doctree_cache: 57 s
-j32 + no caching of _write_doc_doctree_cache: 48 s

Related: sphinx-doc#11124
@marxin marxin mentioned this issue Apr 5, 2023
Merged
@marxin
Copy link
Contributor Author

marxin commented Apr 6, 2023

So the situation with my patch should be back to when multiple jobs are used. When it comes to -j1, yes, the memory usage is higher, but the build finishes sooner. If it's really a problem, I can imagine a sphinx-build option for this optimization. Feel free to re-open the issue if it's the case.

@marxin marxin closed this as completed Apr 6, 2023
@zzzeek zzzeek mentioned this issue Apr 25, 2023
Closed
@AA-Turner
Copy link
Member

@marxin out of interest, what did you use to make the graphs above?

A

@marxin
Copy link
Contributor Author

marxin commented Apr 26, 2023

@marxin out of interest, what did you use to make the graphs above?

Be my quest, it's my matplotlib-based script located here:
https://github.com/marxin/script-misc/blob/master/usage-wrapper.py

@github-actions github-actions bot locked as resolved and limited conversation to collaborators May 27, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@marxin @AA-Turner