slowness

[dfroger]

Fresh and incremental builds are slow on the project I'm working on: about 40 seconds (I would expect < 2 seconds).

The doxyxml/ directory contains 747 files for a total of 49M.

Seems the time is spent is the xml parser, here is the top of the profiling of the DoxygenTypeSubItemFinder.filter_ method:

         41879933 function calls (41753387 primitive calls) in 37.439 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      743    0.013    0.000   37.419    0.050 /home/david/venv/g/lib/python3.6/site-packages/breathe/finder/index.py:39(filter_)
      743    0.007    0.000   37.143    0.050 /home/david/venv/g/lib/python3.6/site-packages/breathe/parser/__init__.py:73(parse)
      743    0.004    0.000   37.113    0.050 /home/david/venv/g/lib/python3.6/site-packages/breathe/parser/compound.py:1064(parse)
      743    0.006    0.000   23.243    0.031 /home/david/.pyenv/versions/3.6.6/lib/python3.6/xml/dom/minidom.py:1954(parse)
      743    0.013    0.000   23.236    0.031 /home/david/.pyenv/versions/3.6.6/lib/python3.6/xml/dom/expatbuilder.py:899(parse)
      743    0.012    0.000   23.204    0.031 /home/david/.pyenv/versions/3.6.6/lib/python3.6/xml/dom/expatbuilder.py:197(parseFile)
     4091    2.722    0.001   23.139    0.006 {method 'Parse' of 'pyexpat.xmlparser' objects}
  1477287    3.613    0.000   14.837    0.000 /home/david/.pyenv/versions/3.6.6/lib/python3.6/xml/dom/expatbuilder.py:736(start_element_handler)
      743    0.003    0.000   13.863    0.019 /home/david/venv/g/lib/python3.6/site-packages/breathe/parser/compoundsuper.py:175(build)
     2229    0.002    0.000   13.854    0.006 /home/david/venv/g/lib/python3.6/site-packages/breathe/parser/compoundsuper.py:184(buildChildren)
      743    0.014    0.000   13.846    0.019 /home/david/venv/g/lib/python3.6/site-packages/breathe/parser/compoundsuper.py:366(build)
    25897    0.052    0.000   13.819    0.001 /home/david/venv/g/lib/python3.6/site-packages/breathe/parser/compoundsuper.py:379(buildChildren)
      344    0.084    0.000    8.117    0.024 /home/david/venv/g/lib/python3.6/site-packages/breathe/parser/compoundsuper.py:2434(build)
   168848    0.115    0.000    7.999    0.000 /home/david/venv/g/lib/python3.6/site-packages/breathe/parser/compoundsuper.py:2442(buildChildren)
    84252    0.197    0.000    7.742    0.000 /home/david/venv/g/lib/python3.6/site-packages/breathe/parser/compoundsuper.py:2511(build)
   233293    0.254    0.000    6.618    0.000 /home/david/venv/g/lib/python3.6/site-packages/breathe/parser/compoundsuper.py:2529(buildChildren)
   233293    0.940    0.000    5.974    0.000 /home/david/venv/g/lib/python3.6/site-packages/breathe/parser/compoundsuper.py:2588(build)
   906428    3.019    0.000    5.114    0.000 /home/david/.pyenv/versions/3.6.6/lib/python3.6/xml/dom/minidom.py:354(__init__)
  1043064    3.255    0.000    4.712    0.000 /home/david/.pyenv/versions/3.6.6/lib/python3.6/xml/dom/expatbuilder.py:274(character_data_handler_cdata)
     1265    0.011    0.000    4.710    0.004 /home/david/venv/g/lib/python3.6/site-packages/breathe/parser/compoundsuper.py:1010(build)
    17161    0.015    0.000    4.691    0.000 /home/david/venv/g/lib/python3.6/site-packages/breathe/parser/compoundsuper.py:1019(buildChildren)
     7948    0.138    0.000    4.608    0.001 /home/david/venv/g/lib/python3.6/site-packages/breathe/parser/compoundsuper.py:1231(build)
   257164    0.140    0.000    4.252    0.000 /home/david/venv/g/lib/python3.6/site-packages/breathe/parser/compound.py:177(buildChildren)
   257164    0.446    0.000    4.113    0.000 /home/david/venv/g/lib/python3.6/site-packages/breathe/parser/compoundsuper.py:1282(buildChildren)
  1233275    1.606    0.000    3.397    0.000 /home/david/venv/g/lib/python3.6/site-packages/breathe/parser/compoundsuper.py:2597(buildChildren)
  1949100    2.844    0.000    2.844    0.000 /home/david/.pyenv/versions/3.6.6/lib/python3.6/xml/dom/minidom.py:994(__init__)
  1154997    2.293    0.000    2.293    0.000 /home/david/.pyenv/versions/3.6.6/lib/python3.6/xml/dom/minidom.py:698(_ensure_attributes)
  1477287    2.185    0.000    2.185    0.000 /home/david/.pyenv/versions/3.6.6/lib/python3.6/xml/dom/minidom.py:678(__init__)
   609257    1.883    0.000    2.141    0.000 /home/david/.pyenv/versions/3.6.6/lib/python3.6/xml/dom/minidom.py:876(_get_attributes)
  2519959    1.090    0.000    1.295    0.000 /home/david/.pyenv/versions/3.6.6/lib/python3.6/xml/dom/minidom.py:284(_append_child)
31657/31634    0.088    0.000    1.085    0.000 /home/david/venv/g/lib/python3.6/site-packages/breathe/parser/compoundsuper.py:1628(build)
  2463108    1.055    0.000    1.055    0.000 {method 'split' of 'str' objects}
    49962    0.497    0.000    1.048    0.000 /home/david/venv/g/lib/python3.6/site-packages/breathe/parser/compoundsuper.py:2744(build)
    20222    0.038    0.000    1.032    0.000 /home/david/venv/g/lib/python3.6/site-packages/breathe/parser/compoundsuper.py:2026(build)
    10020    0.033    0.000    1.029    0.000 /home/david/venv/g/lib/python3.6/site-packages/breathe/parser/compoundsuper.py:1841(build)

Maybe related to issue #420 ? Thanks.

[vermeeren]

Possibly also related to #315, the XML related things are not very efficient currently.

Unlikely to change soon due to lack of contributors, patches always welcome though.

[dfroger]

Thanks for the reply. I'll see if I can submit a patch to improve the perfs.

[dfroger]

Just tested lxml instead of minidom on my 49M of 747 xml files, it's faster. (I have to read #315 where lxml is dicussed.)

minidom

from pathlib import Path
from xml.dom import minidom

for p in Path('doxyxml').glob('*.xml'):
    minidom.parse(str(p))

real	0m12,474s
user	0m12,140s
sys	0m0,332s

lxml

from pathlib import Path
from lxml import etree

for p in Path('doxyxml').glob('*.xml'):
    etree.parse(str(p))

real	0m0,753s
user	0m0,672s
sys	0m0,080s

[dfroger]

Another measurement, parsing 1 file that combine the 747 files:

xsltproc combine.xslt index.xml > all.xml

minidom

real	0m16,656s
user	0m16,196s
sys	0m0,408s

lxml

real	0m0,133s
user	0m0,060s
sys	0m0,028s

[nilsdeppe]

@dfroger this is awesome! Thank you for experimenting with this! Would it be possible to share your changes (or submit a PR)? The project I'm working with currently takes ~90 seconds to build and given your results that could be vastly improved upon 😄

[AnthonyDiGirolamo]

If anyone is still interested in lxml I started a PR for it #891 It's not complete but unit tests are passing.