Option to make docs build in parallel

[isaacgsmith]

Description
See https://www.sphinx-doc.org/en/master/man/sphinx-build.html#cmdoption-sphinx-build-j

Now, make html allows a CORES argument to let you build the documentation in parallel. In the GitHub pipelines, we use CORES=auto which makes the documentation build with all available cores.

Motivation and context

The documentation build is very slow.

How has this been tested?

• Testing pipeline.
• Other.
  Documentation built locally and on GitHub.

Examples

Compare https://github.com/tardis-sn/tardis/actions/runs/1381526801 with https://github.com/smithis7/tardis/actions/runs/1387845143.

See the note: https://smithis7.github.io/tardis/branch/speed_up_doc/development/documentation_guidelines.html

Type of change

• Bug fix.
• New feature.
• Breaking change.
• None of the above.

Checklist

• My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • (optional) I have built the documentation on my fork following the instructions.
• I have assigned and requested two reviewers for this pull request.

[tardis-bot]

Before a pull request is accepted, it must meet the following criteria:

• Is the necessary information provided?
• Is this a duplicate PR?
  • If a new PR is clearly a duplicate, ask how this PR is different from the original PR?
  • If this PR is about to be merged, close the original PR with a link to this new PR that solved the issue.
• Does it pass existing tests and are new tests provided if required?
  • The test coverage should not decrease, and for new features should be close to 100%.
• Is the code tidy?
  • No unnecessary print lines or code comments.

[tardis-bot]

Before a pull request is accepted, it must meet the following criteria:

• Is the necessary information provided?
• Is this a duplicate PR?
  • If a new PR is clearly a duplicate, ask how this PR is different from the original PR?
  • If this PR is about to be merged, close the original PR with a link to this new PR that solved the issue.
• Does it pass existing tests and are new tests provided if required?
  • The test coverage should not decrease, and for new features should be close to 100%.
• Is the code tidy?
  • No unnecessary print lines or code comments.

[tardis-bot]

Before a pull request is accepted, it must meet the following criteria:

• Is the necessary information provided?
• Is this a duplicate PR?
  • If a new PR is clearly a duplicate, ask how this PR is different from the original PR?
  • If this PR is about to be merged, close the original PR with a link to this new PR that solved the issue.
• Does it pass existing tests and are new tests provided if required?
  • The test coverage should not decrease, and for new features should be close to 100%.
• Is the code tidy?
  • No unnecessary print lines or code comments.

[codecov]

Codecov Report

Merging #1803 (3755d75) into master (60e044e) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #1803   +/-   ##
=======================================
  Coverage   58.22%   58.22%           
=======================================
  Files          66       66           
  Lines        6722     6722           
=======================================
  Hits         3914     3914           
  Misses       2808     2808           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 60e044e...3755d75. Read the comment docs.

[epassaro]

Hi @smithis7, sorry for not reviewing this lately.

I think make allows you to do something like this:

1. Read a variable from the shell environment
2. If that variable is not set, use a default value

That variable could be called N_SPHINX_JOBS or something like that, and stored in .bashrc or any shell configuration file. In this way, users are not forced to use all their cores to build the documentation.

What do you think? @jaladh-singhal @wkerzendorf

[isaacgsmith]

Hi @smithis7, sorry for not reviewing this lately.

I think make allows you to do something like this:

1. Read a variable from the shell environment
2. If that variable is not set, use a default value

That variable could be called N_SPHINX_JOBS or something like that, and stored in .bashrc or any shell configuration file. In this way, users are not forced to use all their cores to build the documentation.

What do you think? @jaladh-singhal @wkerzendorf

We can see what they think -- I feel like in general though, people building the documentation locally would like it to be built fast and are not as concerned about their cores for the 10-15 minutes that the build is occurring. If someone is really concerned about their cores, they are always able to type in the full sphinx command sphinx-build -b html -d _build/doctrees . _build/html.

[epassaro]

@smithis7 Can you post the result of runs with cores 1 and auto and check if the difference is noticeable?

[isaacgsmith]

@epassaro Two recent builds:
https://github.com/tardis-sn/tardis/runs/4142758941?check_suite_focus=true
and
https://github.com/smithis7/tardis/runs/4142812289?check_suite_focus=true

CORES=auto takes roughly 4:30 off the build step. It does even more locally.

Additionally, this difference will become greater as we add more notebooks.

[epassaro]

@epassaro Two recent builds: https://github.com/tardis-sn/tardis/runs/4142758941?check_suite_focus=true and https://github.com/smithis7/tardis/runs/4142812289?check_suite_focus=true

CORES=auto takes roughly 4:30 off the build step. It does even more locally.

Additionally, this difference will become greater as we add more notebooks.

I see, great work!

I've implemented caching the conda env in carsus pipelines, that should cut other ~3 mins.