MNT use a single job by default with sphinx build

[adrinjalali]

I don't think in most cases parallel run is going to be faster, at least for me the difference is about 6 minutes, sequencial run being faster:

# auto
4765.77user 2308.57system 27:44.35elapsed 425%CPU (0avgtext+0avgdata 3165084maxresident)k
14448inputs+2382992outputs (198major+9639491minor)pagefaults 0swaps

# 1
3799.71user 1916.26system 21:54.21elapsed 434%CPU (0avgtext+0avgdata 3157024maxresident)k
22064inputs+764960outputs (69major+4528302minor)pagefaults 0swaps

This PR makes sequencial run the default for sphinx.

cc @lesteve @glemaitre

[glemaitre]

Yep this seems reasonable then.

[lesteve]

Can you add the sphinx version you used in your top post for future reference?

If we think SPHINX_NUMJOBS=1 is always a good idea, we should probably remove export SPHINX_NUMJOBS=2 from build_tools/circle/build_doc.sh and possibly remove SPHINX_NUMJOBS=1 from .circle/config.yml since it will default to 1.

[lesteve]

My data point on my machine with sphinx 6.1.3 for the make html-noplot part (i.e. without running the examples):

SPHINX_NUMJOBS=1 make html-noplot 2m52s
SPHINX_NUMJOBS=2 make html-noplot 2m6s
SPHINX_NUMJOBS=auto make html-noplot 1m29s (same thing as SPHINX_NUMJOBS=4, I have 2 physical cores and 4 logical cores on my machine)

so in this case SPHINX_NUMJOBS seems a bit useful, roughly from 3 minutes to 1.5 minute with SPHINX_NUMJOBS=auto.

IIRC examples are not run in parallel by sphinx-gallery, so I don't expect SPHINX_NUMJOBS to provide much speed-up when running the examples.

I would say people are more likely to do make html-noplot on their local machine than make html so maybe SPHINX_NUMJOBS=auto is still a bit useful locally?

[adrinjalali]

I personally almost never remember to do noplots, and I just run make html, which is what we tell people to do in our docs. And I'm pretty sure the number of jobs has an effect there. My spihnx versoins:

  sphinx                         6.1.3         pyhd8ed1ab_0         conda-forge
  sphinx-gallery                 0.12.1        pyhd8ed1ab_0         conda-forge
  sphinx-prompt                  1.4.0         pyhd8ed1ab_0         conda-forge
  sphinxcontrib-applehelp        1.0.4         pyhd8ed1ab_0         conda-forge
  sphinxcontrib-devhelp          1.0.2         py_0                 conda-forge
  sphinxcontrib-htmlhelp         2.0.1         pyhd8ed1ab_0         conda-forge
  sphinxcontrib-jsmath           1.0.1         py_0                 conda-forge
  sphinxcontrib-qthelp           1.0.3         py_0                 conda-forge
  sphinxcontrib-serializinghtml  1.1.5         pyhd8ed1ab_2         conda-forge
  sphinxext-opengraph            0.8.1         pyhd8ed1ab_0         conda-forge

[adrinjalali]

Since now we're setting the number of jobs in circleci config, removing it from the script makes sense.

[lesteve]

I tried running locally the full doc build and I get very similar timings with both SPHINX_NUMJOBS=1 and SPHINX_NUMJOBS=auto, roughly 20 minutes. I used the same environment as in the CI:

conda-lock install --name scikit-learn-doc build_tools/circle/doc_linux-64_conda.lock

The sphinx versions are:

sphinx                    6.0.0              pyhd8ed1ab_2    conda-forge
sphinx-gallery            0.11.1             pyhd8ed1ab_0    conda-forge
sphinx-prompt             1.4.0              pyhd8ed1ab_0    conda-forge
sphinxcontrib-applehelp   1.0.4              pyhd8ed1ab_0    conda-forge
sphinxcontrib-devhelp     1.0.2                      py_0    conda-forge
sphinxcontrib-htmlhelp    2.0.0              pyhd8ed1ab_0    conda-forge
sphinxcontrib-jsmath      1.0.1                      py_0    conda-forge
sphinxcontrib-qthelp      1.0.3                      py_0    conda-forge
sphinxcontrib-serializinghtml 1.1.5              pyhd8ed1ab_2    conda-forge
sphinxext-opengraph       0.7.5                    pypi_0    pypi

I personally almost never remember to do noplots, and I just run make html, which is what we tell people to do in our docs.

The first thing we mention is make which does make html-noplot so you don't have to remember much 😉. Having said that, I did not remember that make was doing make html-noplot probably because it was introduced after I learned to do make html-noplot ...

All in all, on my machine it seems like SPHINX_NUMJOBS=auto is useful for make html-noplot and does not improve make html but does not make it worse contrary to what you are seeing on your machine. So I am not sure what to do ...

Can you try if make html-noplot is also worse with SPHINX_NUMJOBS=auto on your machine? If that's the case, I am fine merging this PR.

[adrinjalali]

make html-noplot is faster for me too, but that's only because we don't run any code for that.

In terms of tradeoffs, this PR saves me 6 minutes on each run, and makes you wait one extra minute compared to status quo 😁

Ideally we'd be able to control the number of jobs for sphinx-gallery and set that to 1, while leaving the rest on auto.

[lesteve]

OK let's merge this one, in an ideal world we would keep SPHINX_NUMJOBS=auto during make html-noplot but I don't have the energy to figure out how to do it in our Makefile.

Ideally we'd be able to control the number of jobs for sphinx-gallery and set that to 1, while leaving the rest on auto.

Again, examples are not run in parallel, I watched the htop output while running make html and this matches my expectation. Also this very old sphinx-gallery issue sphinx-gallery/sphinx-gallery#25.