cpp: Slow reference resolution #10966
Comments
|
Ok, so I think the root cause is that we've got ~1000 functions in top-level namespace and there's If I select one symbol ( So it's about calling 3000x (3x likely due naming versions?) |
|
The same problem happens also when |
|
I have an old branch with optimizations that I have rebased: https://github.com/jakobandersen/sphinx/tree/c_cpp_optimizations |
|
Oh, great! I can confirm it helps, now I get to |
|
Memoizing calls to |
|
@jbms, indeed, and if I remember correctly I also have some code fragments in that direction. Though, it's a bit involved to change, e.g., due to lookup in anon entities. I'll try do dig in my branches and see what comes up. |
|
@jakobandersen: Can you please make a pull request that includes the patches addressing this issue? |
|
Note such a PR wouldn't be included in Sphinx 6.0.0, it would go into 6.1 or later. A |
|
@jakobandersen: Can you please make a submit request with the changes now? |
|
When building the Linux kernel documentation, over 12 minutes are spent in |
Hi @jbms -- happy to have a look at this, do you have any pointers for where the lookups happen most frequently? I'm not that familiar with the C++ domain. A |
|
@vegard -- we recently merged a few performance improvements, would you be able to test with the latest unreleased Sphinx? A |
|
@AA-Turner I haven't looked at this in a while, but the best way to investigate would be to build something that is slow, and use the Python profiler to figure out where the time is spent (as @marxin did). There are a few things to keep in mind:
In general, the problem is all of the various lookup/find methods in the C and C++ domains, which currently, within a given scope, just iterate over every symbol and check if it matches. This becomes slow if the scope contains a large number of symbols. |
|
I haven't had time to look at this for while either, but the resurgence of the issue sparked some concrete ideas for how to implement the speedups. I'll try to put something together soon. |
I tested with sphinx-build 7.2.6 (latest when I do plain with sphinx-build 7.3.0+/91ed62272712 (using your command): These are just two individual runs and there is always a bit of variation, so these look pretty much the same to me. |
I'm not sure what the tags feature in C is, but it seems like it might work to just have a single implementation (basically the C++ domain with the addition of macros) with a shared symbol table, but with some branches to handle differences between C and C++, e.g. possibly limiting what can be parsed depending on language and not mangling symbols for C. |
|
Here is some related work I have done to speed up the C domain: https://github.com/donaldh/sphinx/tree/c-domain-speedup |
|
PR is welcome ! (though I'm not sure when I'll have for this since I'm not that familiar with the C/C++ domain) |
I tried this out building zephyr docs which I found ran into this exact issue. The time to build went from ~420s to ~240s for me which is a significant improvement. @donaldh please PR it! |
|
@teburd The C domain work is merged. |
Any hints as to when this might be made available in a release? We could really use the significant speed improvement for Zephyr docs :) |
Mmh, we don't really have a regular release policy (mainly due to the fact that Adam is the only one that can do it I think) but if you really want to have it included in your docs, you can install the main branch instead of the released one using I think you can also add that in the requirements file: (I didn't test the commands but it should be something like that I assume) |
Describe the bug
We've just ported GCC documentation to Sphinx (thank you guys for the tool). However, I noticed one manual build 45s (
90swith cProfile enabled) on my machine and I create an upstream bug:https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107613
We've got about 1000 cpp:function definitions. Is it an expected build time or a bug in Sphinx?
@jakobandersen
How to Reproduce
cProfile report:
Environment Information
5.3
Sphinx extensions
No response
Additional context
No response
The text was updated successfully, but these errors were encountered: