Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cpp: slow sphinx-build read when -j $big is used #10969

Closed
marxin opened this issue Nov 11, 2022 · 2 comments
Closed

cpp: slow sphinx-build read when -j $big is used #10969

marxin opened this issue Nov 11, 2022 · 2 comments

Comments

@marxin
Copy link
Contributor

marxin commented Nov 11, 2022

Describe the bug

Originally thought it's #10967, but the suggested patch hasn't helped me.
The problem is that reading sources... takes ages (~15 minutes and more) if I use -j 160:

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    497/1    0.001    0.000  311.019  311.019 {built-in method builtins.exec}
        1    0.000    0.000  311.019  311.019 sphinx-build:1(<module>)
        1    0.000    0.000  310.746  310.746 build.py:306(main)
        1    0.004    0.004  310.745  310.745 build.py:268(build_main)
        1    0.000    0.000  310.399  310.399 application.py:339(build)
        1    0.000    0.000  310.399  310.399 __init__.py:301(build_update)
        1    0.000    0.000  310.398  310.398 __init__.py:314(build)
        1    0.000    0.000  310.395  310.395 __init__.py:384(read)
        1    0.009    0.009  310.386  310.386 __init__.py:456(_read_parallel)
      324    0.044    0.000  310.023    0.957 parallel.py:120(_join_one)
      293    0.008    0.000  307.161    1.048 __init__.py:476(merge)
      293    0.050    0.000  273.810    0.935 __init__.py:351(merge_info_from)
      293    0.009    0.000  273.521    0.934 cpp.py:7917(merge_domaindata)
101038/293    1.031    0.000  273.512    0.933 cpp.py:4807(merge_with)
   618010   59.922    0.000  261.630    0.000 cpp.py:4412(_find_named_symbols)
131399834   61.641    0.000  184.644    0.000 cpp.py:4440(matches)
        1    0.005    0.005  167.875  167.875 parallel.py:102(join)
      311    0.239    0.001  142.469    0.458 parallel.py:88(add_task)
131399833   96.795    0.000  123.003    0.000 cfamily.py:84(__eq__)
      586   33.457    0.057   33.643    0.057 {built-in method _pickle.loads}
131500914   16.998    0.000   16.998    0.000 cpp.py:4462(candidates)
131408156/131408152   14.050    0.000   14.050    0.000 {built-in method builtins.getattr}
131412043   12.171    0.000   12.171    0.000 {method 'items' of 'dict' objects}
   201490    0.156    0.000   10.658    0.000 cpp.py:4050(get_newest_id)

How to Reproduce

$ git clone --depth 1 git://gcc.gnu.org/git/gcc.git
$ cd gcc
$ time sphinx-build -b texinfo -d doctrees /home/marxin/Programming/gcc/gcc/doc/gccint output -j 160

Environment Information

5.3

Sphinx extensions

No response

Additional context

No response

xionghul pushed a commit to xionghul/gcc that referenced this issue Nov 11, 2022
Noticed that the documentation build can stuck on a machine with
many cores (160) and I identified a real sphinx problem:
sphinx-doc/sphinx#10969

Note the parallel can help just for some manuals and it is not critical
for us.

ChangeLog:

	* doc/Makefile: Disable -j auto.
@jakobandersen
Copy link
Contributor

I believe this is a duplicate of #10966, as in parallel mode the symbol tables needs to be merged, which requires lookup.

@marxin
Copy link
Contributor Author

marxin commented Apr 4, 2023

Closing as duplicate.

@marxin marxin closed this as completed Apr 4, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators May 5, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

2 participants