Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for executing notebooks in parallel during JupyterBook Build #1735

Open
nishikantparmariam opened this issue May 23, 2022 · 1 comment
Labels
enhancement New feature or request

Comments

@nishikantparmariam
Copy link

nishikantparmariam commented May 23, 2022

Context

If there are big and many notebooks, then jb build . can take too much time. Specially when building notebooks is a part of a recurring automated process or a CI pipeline. This may be improved by executing notebooks in parallel.

There should be a way to specify the number of notebooks to be executed in parallel (default 1) or the number of job that can be used to execute the notebooks, say using jb build . -j 5. I could not find a way to do this currently.

Please let me know if there is an existing solution / update / workaround for this. Else, I would like to work on PR as per below proposal.

Proposal

There is a similar issue - issues#1719

Based on initial findings -

  • MyST-NB already supports parallel execution as per this changelog.
  • Introducing an optional argument j or job, usage - jb build . -j 5.
  • build_sphinx already supports the job argument, hence we can pass the job argument from here as well.

Tasks and updates

No response

@nishikantparmariam nishikantparmariam added the enhancement New feature or request label May 23, 2022
@nishikantparmariam nishikantparmariam changed the title Supporting executing notebooks in parallel during JupyterBook Build Support for executing notebooks in parallel during JupyterBook Build May 23, 2022
@migueltorrescosta
Copy link

migueltorrescosta commented May 21, 2024

Temporary solution: https://github.com/executablebooks/jupyter-cache

Just started using jupyter-book ( amazing project 🙌 ), and this feature would make it a lot better: iterating on the web generated pages locally is very slow due to this constraint. It is possible to work around them with notebook_execute: off with nbconvert, or by only including the notebook I am currently working on, however no solution is as smooth as allowing the parallel execution of notebooks.

Is there something I can do to help with this feature?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants