Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI Pipeline + test framework for Jupyter Notebooks #19

amit1rrr opened this Issue Mar 29, 2019 · 1 comment


None yet
1 participant
Copy link

amit1rrr commented Mar 29, 2019


Jupyter Notebooks neither have a dedicated test framework nor a CI pipeline. Which results in,

  • Almost no tests written for Notebooks
  • Lack of confidence whether a Notebook would run as expected when executing on a fresh kernel
  • Inability to use Notebooks for any production workflow

Users tend to jump around Notebook cells while developing, resulting in an unpredictable kernel state. Given Notebooks are prone to muddled state, importance of testing & continuous integration increases multifold.


Here are the bare minimum things we need,

  • Every time a Notebook change is committed, run all cells from top to bottom and make sure no cell execution results in error
  • Ability to write and execute unit tests for Notebook code
  • Given a set of inputs, provide ability to specify & assert that the actual outputs of certain Notebook cells matches the expected outputs

All of the above is possible today by tinkering with separate tools such as Doctest, unittest, papermill etc. We need to combine it all, fill the gaps and make an open source testing framework dedicated to testing Jupyter Notebooks.

As a last step, this testing framework can then be integrated into ReviewNB interface to run tests after each commit is pushed for users who wish to enable CI pipeline on their repos.

Feel free to upvote/downvote the issue indicating whether you think this is useful feature or not. I also welcome additional questions/comments/discussion on the issue.


This comment has been minimized.

Copy link
Contributor Author

amit1rrr commented Apr 5, 2019

The test framework is now available here:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.