Notebook Testing Feedback

[alex-treebeard]

Hey folks (specifically @balopat),

I noticed you tried using treebeard for testing example notebooks and that it hasn't exactly gone smoothly (context).

Just wanted to encourage you to leave me a list of problems, frustrations, suggestions, and any other feedback I can use for prioritising work on my end (here or in gitter/issues).

[balopat]

Hey @alex-treebeard, I like the reachout :) I like treebeard. But then the specific use case here convinced me that it wasn't a fit (at least for now), so I ended up rolling my own driver for the notebook tests. Specifically:

• ability to run notebooks in their own virtual environment - most of our users use notebooks in Colab and currently we install cirq via !pip install cirq in the first cells. This path needs to be tested.
• I typically want all of our notebooks to be tested and then exclude some for the right reasons using glob patterns. Currently treebeard only supports union of globs which is less intuitive.
• not sure about parallelization of the notebooks - I haven't checked. But using xdist seems to work okay for now, and will allow me the flexibility to shard the notebook runs even further, on the Github Action level, to get more speedup. At the current stage the timing is 11 minutes which is already too long.

Other than that treebeard was super easy to get started and provided a very pretty output, that would have been very useful. I might still use it for the "jupyter user journey" that we want to support: when the user checks out the repo and uses the cirq code itself in a Jupyter notebook.

[alex-treebeard]

Thanks so much for the writeup.

I can see how much work you put into this testing setup so I'd like to get your feedback in a few days after I put in some fixes on my end as I reckon this is a great case study for my project.

Will follow up on your fork.