parsing performance worsened

[svigerske]

With larger .dox files, the time to parse the file seems to have increased considerably between Doxygen 1.8.14 and 1.8.15.

With the 100KB .dox file in datalib.dox.zip, I get with doxygen 1.8.14:

real	0m0.169s
user	0m0.133s
sys	0m0.027s

But with doxygen 1.8.15 this becomes:

real	0m1.333s
user	0m1.300s
sys	0m0.026s

On another 490KB file (similar structure), time even increases from 0.9 seconds to 112 seconds.

[albert-github]

Please include your Doxyfile as well as I get:

warning: Found unknown command ``\gmscodeinclude'

also run with -d time (so doxygen -d time Doxyfile and of course QUIET=NO) and include the resulting, terminal, output for 1.8.14 and 1.8.15.

[svigerske]

Well, I was running with an empty Doxyfile, ignored all warnings, and could see that most time is spend on the "Parsing file..." line.

But here a directory with .dox file (with \gmscodeinclude removed), Doxyfile, and the logs from 1.8.14 and 1.8.15: bigdox.zip

1.8.14 says

Spent 0.057 seconds in Parsing files

1.8.15 says

Spent 1.071 seconds in Parsing files

[mosra]

Possibly related: while reporting #6715 I noticed that #6592 (commit 07ae32a) increases the resulting Doxygen binary size by 200 kB in Release build (or, conversely, if I revert that commit to work around the bug, Doxygen installation shrinks by 200 kB). That sounds like a lot for a seemingly simple parser change and I suspect this might have some measurable perf impact.

[albert-github]

I just did a quick check and reverted the #6592 patch (was easy as it were only 2 lines changed without interactions in this case) and the timing stayed at the same high level.
Looking with -d time -d lex I saw that in the mentioned part the commentscan.l is called, so this lexer or a function called by it is a more likely candidate.

[albert-github]

Change in time is a result of the improvement of the markdown processing in #674, the markdown processor is now at a better place and more logical place.

[doxygen]

The move of the markdown processing does not yet explain the increased total running time (markdown is also done in the 1.8.14 version). So it is good to do some code profiling.

[albert-github]

I see a lot of pages (by head over 120) in the "bigdox" case and it looks like they are fed one by one to markdown processor.
Might some startup of the markdown processor / caching cause the problem? or maybe the "entry" handling (marshalling)?
I looked at the direct differences in the markdown processor but could not find a direct cause.

[doxygen]

In this commit c61d870 I've moved the markdown processing out of the parseCommentBlock() loop, which should bring the performance back to the 1.8.14 level.

[doxygen]

This issue was previously marked 'fixed but not released',
which means it should be fixed in doxygen version 1.8.16.
Please verify if this is indeed the case. Reopen the
issue if you think it is not fixed and please include any additional information
that you think can be relevant (preferably in the form of a self-contained example).

[svigerske]

Looks good. With 1.8.16 for the datalib.dox file posed at the beginning I am now back to

real	0m0.175s
user	0m0.092s
sys	0m0.031s

Thank you!