testing Jupyter notebooks on Binder without human interaction

[villaa]

Since much of our user base is envisioned to use these learning materials through binder it would be good to set up automated tests to test whether the notebooks work within the binder environment.

Technically our first step would just be to figure out what Jupyter environment binder is adding to our yml environment, @gerudo7 is looking into that...

But ideally we would actually send the notebooks out from Travis-CI to be run in the binder environment, so they would fail if binder itself changes their environment, as they have in the past.

[nuclearGoblin]

I've found an issue on the mybinder repo and an organization related to testing in binder automatically, but I'm not sure how to implement it yet.

[nuclearGoblin]

Okay, spent some time re-reading the previous links and I think they're more about testing Binder itself than testing if things run within Binder. found more info on the binder discussion board: https://discourse.jupyter.org/t/binder-as-part-of-a-test-framework/235/2

We started work on building examples of using repo2docker as part of your CI pipeline in https://github.com/binder-examples/continuous-build/ 9. It is a pretty sophisticated (some would say complex) setup right now. I think having simpler ones would be great as that would help people to get started.
A super simple thing I have started doing is to run repo2docker . papermill a-notebook.ipynb in a repository that has notebooks that should run on mybinder.org and I want to (somewhat) automatically check if they run to completion.

From what I'm understanding - we can provide a dockerfile to give Binder stricter parameters. Is that something we want to do? @villaa . I think we can just as easily provide strict version numbers, as a weird interaction between package versions turned out to be the issue last time, and that will probably prevent it from running into problems in the future. But, creating a dockerfile would be a bit more "airtight."

[villaa]

From what I'm understanding - we can provide a dockerfile to give Binder stricter parameters. Is that something we want to do? @villaa . I think we can just as easily provide strict version numbers, as a weird interaction between package versions turned out to be the issue last time, and that will probably prevent it from running into problems in the future. But, creating a dockerfile would be a bit more "airtight."

@gerudo7 I would be open to either method. Perhaps starting with specifying exact versions that don't conflict is a good set to know. Then perhaps a new issue opened for creating the docker file could be a next-level improvement.

[nuclearGoblin]

Cool. In that case - closing this because it doesn't seem directly possible, and I'll link back to this issue in a new one about creating a dockerfile.