Investigate use of testbook for testing Jupyter notebooks

[matthewfeickert]

Description

At SciPy 2020 @rohitsanj gave a (very good) lightning talk on nteract's testbook which is used for unit testing Jupyter Notebooks. Given that Carol Willing mentioned it is designed to be complimentary with papermill it might be worth seeing if it can be used in our testing of the example notebooks in CI.

cc @phinate as neos uses lots of notebooks.

[rohitsanj]

Hi @matthewfeickert, thanks for the mention! We would be more than excited to see testbook be used by neos or any other repo.

If you'd like, I could try writing some of the tests using testbook for any notebook that you can point out - win win situation 😄

Thanks again!

cc @MSeal

[matthewfeickert]

If you'd like, I could try writing some of the tests using testbook for any notebook that you can point out

@rohitsanj fantastic! Thanks for the incredibly generous offer.

I'll first describe how we use papermill in our CI to get your thoughts on where testbook might be the best fit. All of our examples in our documentation are Jupyter notebooks rendered with Sphinx. We use papermill to run all of these notebooks in our CI each PR to ensure that we never break our examples. As some of our examples involve quite a bit of computation this can be slow. Maybe it would make more sense to try and unit test all the functions defined in the notebooks in CI to speed things up and then just run the full papermill tests on a nightly schedule job?

If you think this makes sense, maybe a good place to start is our simplest example notebook: hello-world.ipynb

[MSeal]

Btw for context, papermill is good for integration testing, testbook is good for unittesting. If you have notebooks with high branching factor or want to avoid expensive computation associated with functions with making dummy parameters, testbook should be a good choice. So I think you have the right idea of how testbook might fit in.

[matthewfeickert]

or want to avoid expensive computation associated with functions with making dummy parameters

Yeah, at the moment we're more just running the notebooks through to make sure the public APIs don't break and checking a few values with scrapbook, but the notebook CI takes about 17 minutes at the moment.

[MSeal]

At Netflix we reused our scheduler in a dedicated deployment to parallelize out all the papermill integration tests, which helped with integration test wall time substantially.

Unfortunately GitHub actions doesn't have higher core options yet: https://github.community/t/higher-cpu-count-virtual-environments-planned/16338 so you do get stuck with the mostly serial execution times.