Performance problem: Is there a way to make doxygen parallel computing improve performance?

[BookerShi]

Describe the bug
Performance problem: We have a project here that needs to generate .dot files. The file size is about 1.3GB. I found that using doxygen to generate dot files is very slow and takes several days. Is there a way to make doxygen parallel computing improve performance?

Version
doxygen 1.9.7

Additional context
my configuration is CALL_GRAPG = YES , HAVE_DOT=YES, DOT_CLEANUP=YES, RECURSIVE=YES

[albert-github]

• Which version of doxygen are you using exactly, the 1.9.7 version is the master version and thus a moving target. Required is doxygen -v).
• please add the output of the command doxygen -x Doxyfile to your question (preferably in a, compressed, tar or zip file) so we are sure of the settings used (as in teh question there is already a typing error: CALL_GRAPG should be CALL_GRAPH.
• which OS are you using and how many physical cores do you have?

You say: " The file size is about 1.3GB." Is your project just a single file? or is this the size of the source directory or the size of the output?

From the, scarce, information I get you are generating the HTML and LaTeX documentation, is this intended or do you only use HTML?

The first run / run on a clean output directory will take a bit longer, subsequent runs on the already existing directory will be faster (due to some caching by means of the generated md5 / output files).

[BookerShi]

Thank you for your reply, my project is not just a file, it is a directory including souce code and other dependencies. My server has 8 cores, I am not sure whether doxygen have parallel operation that can speed up the .dot time of generation.
I also have a question, Adding "-x" in the command "doxygen -x Doxyfile" means speeding up the second generating time?

[albert-github]

No doxygen -x Doxyfile is just for getting some information that we need so we are 100% sure what you use as settings.

There are options regarding speed:

• DOT_NUM_THREADS for executing some dot threads in parallel, should be set OK but depending on your system comfiguration a setting of 8 (or 4 in case you cores are 4 physical core and 4 hyper thread cores) might be more appropriate
• NUM_PROC_THREADS for some not-dot related parallel processing

[doxygen]

How many dot files is doxygen generating for your project?

Which OS are you using? Windows is known to be much lower than Linux or MacOS.

Setting DOT_MULTI_TARGETS to YES may also help a bit.

[BookerShi]

Thank for your positive reply. I appreciat it. I've done some experiments through changing some terms you mentioned. I found the result of speeding up is not that good.My configuration is following:

…

______________________________________ LOOKUP_CACHE_SIZE      = 4 EXTRACT_ALL            = YES EXTRACT_PRIVATE        = YES EXTRACT_PACKAGE        = YES EXTRACT_STATIC         = YES EXTRACT_LOCAL_METHODS  = YES INPUT                  = /home/spj/libxml2-2.9.10 RECURSIVE              = YES GENERATE_LATEX         = NO HAVE_DOT               = YES DOT_NUM_THREADS        = 8 TEMPLATE_RELATIONS     = YES CALL_GRAPH             = YES CALLER_GRAPH           = YES GRAPHICAL_HIERARCHY    = NO DIRECTORY_GRAPH        = NO DOT_MULTI_TARGETS      = YES DOT_CLEANUP            = NO

______________________________________ My experiments results is here: INPUT       TIME   DOT_NUM_THREADS  NUM_PROC_THREADS  GENERATE_LATEX  LOOKUP_CACHE_SIZE  DOT_MULTI_TARGETS  libxml(34.8MB)         23s            0                  1                 no               0                      YES                              22.56s         0                 16                 no               0          YES              24.85s         16                 0                 no               0                      YES              24.19s         16                 16                 no               0   YES              24.35s         16                 16                 no               3   YES              36.81s         16                 16                 yes               3   YES  In addtion, My core number is 20, I set DOT_NUM_THREADS as 16 Could you give me some suggestions about this result? 

------------------ 原始邮件 ------------------ 发件人: "doxygen/doxygen" ***@***.***>; 发送时间: 2023年4月20日(星期四) 下午4:23 ***@***.***>; ***@***.******@***.***>; 主题: Re: [doxygen/doxygen] Performance problem: Is there a way to make doxygen parallel computing improve performance? (Issue #9989) No doxygen -x Doxyfile is just for getting some information that we need so we are 100% sure what you use as settings. There are options regarding speed: DOT_NUM_THREADS for executing some dot threads in parallel, should be set OK but depending on your system comfiguration a setting of 8 (or 4 in case you cores are 4 physical core and 4 hyper thread cores) might be more appropriate NUM_PROC_THREADS for some not-dot related parallel processing — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[BookerShi]

Also , I just wanna gennerating function call graph using my program, and only using the dot file not latex and other files, which options in the Doxyfile can be removed in order to speed up the generating time?

…

------------------ 原始邮件 ------------------ 发件人: "1280960051" ***@***.***>; 发送时间: 2023年4月23日(星期天) 晚上8:16 ***@***.***>; 主题: 回复： [doxygen/doxygen] Performance problem: Is there a way to make doxygen parallel computing improve performance? (Issue #9989) Thank for your positive reply. I appreciat it. I've done some experiments through changing some terms you mentioned. I found the result of speeding up is not that good.My configuration is following:

______________________________________ LOOKUP_CACHE_SIZE      = 4 EXTRACT_ALL            = YES EXTRACT_PRIVATE        = YES EXTRACT_PACKAGE        = YES EXTRACT_STATIC         = YES EXTRACT_LOCAL_METHODS  = YES INPUT                  = /home/spj/libxml2-2.9.10 RECURSIVE              = YES GENERATE_LATEX         = NO HAVE_DOT               = YES DOT_NUM_THREADS        = 8 TEMPLATE_RELATIONS     = YES CALL_GRAPH             = YES CALLER_GRAPH           = YES GRAPHICAL_HIERARCHY    = NO DIRECTORY_GRAPH        = NO DOT_MULTI_TARGETS      = YES DOT_CLEANUP            = NO

______________________________________ My experiments results is here: INPUT       TIME   DOT_NUM_THREADS  NUM_PROC_THREADS  GENERATE_LATEX  LOOKUP_CACHE_SIZE  DOT_MULTI_TARGETS libxml(34.8MB)         23s            0                  1                 no               0                      YES                              22.56s         0                 16                 no               0          YES              24.85s         16                 0                 no               0                      YES              24.19s         16                 16                 no               0   YES              24.35s         16                 16                 no               3   YES              36.81s         16                 16                 yes               3   YES  In addtion, My core number is 20, I set DOT_NUM_THREADS as 16 Could you give me some suggestions about this result?

------------------ 原始邮件 ------------------ 发件人: "doxygen/doxygen" ***@***.***>; 发送时间: 2023年4月20日(星期四) 下午4:23 ***@***.***>; ***@***.******@***.***>; 主题: Re: [doxygen/doxygen] Performance problem: Is there a way to make doxygen parallel computing improve performance? (Issue #9989) No doxygen -x Doxyfile is just for getting some information that we need so we are 100% sure what you use as settings. There are options regarding speed: DOT_NUM_THREADS for executing some dot threads in parallel, should be set OK but depending on your system comfiguration a setting of 8 (or 4 in case you cores are 4 physical core and 4 hyper thread cores) might be more appropriate NUM_PROC_THREADS for some not-dot related parallel processing — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[albert-github]

@BookerShi
Note that attachments etc. are not at all or not properly shown when replying by email so the comment #9989 (comment) doesn't contain useful information

Regarding the second email, did you try to set GENERATE_LATEX=NO? Info looks like to be embedded in the hard to read / unreadable comment #9989 (comment) comment

[BookerShi]

My results using doxygen -x Doxyfile are as follows： Since I only want to use procedurally generated dot files, I have two ideas to speed up. One is to do parallel acceleration, that is, multithreading, which should be faster since it is running later on the server. However, the current experimental results show that increasing the value of DOT_NUM_THREADS does not have high speed improvement. I would like to ask you what might be the cause of this? My second idea is to delete configuration items unrelated to dot file generation to improve the speed. For example, I only use dot files in html, not latex content, so I disable GENERATE_LATEX, which helps to improve the speed, but it is not enough. So，What other options can be disabled to increase speed?

…

------------------ 原始邮件 ------------------ 发件人: "doxygen/doxygen" ***@***.***>; 发送时间: 2023年4月23日(星期天) 晚上8:42 ***@***.***>; ***@***.******@***.***>; 主题: Re: [doxygen/doxygen] Performance problem: Is there a way to make doxygen parallel computing improve performance? (Issue #9989) @BookerShi Note that attachments etc. are not at all or not properly shown when replying by email so the comment #9989 (comment) doesn't contain useful information Regarding the second email, did you try to set GENERATE_LATEX=NO? Info looks like to be embedded in the hard to read / unreadable comment #9989 (comment) comment — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>