identify strategy for auto-testing tutorials

[gregcaporaso]

nbconvert may be useful here for converting to other formats, testing, etc. They have rst/ipynb conversion tools in that package.

[jairideout]

This would be awesome to get in place for our current notebook tutorials, and then convert the rest of the tutorials after that. Right now it's really easy to break the tutorials when script names/options change (e.g., #1298), and it sucks to have to manually run the tutorials to make sure they're still working.

One hack that might work is to use nbconvert to convert to python, and then have some custom testing code execute the python file to check for errors.

A better way: have code that connects to an ipython kernel and runs all code cells in a notebook (as-is, without any conversion to another format). AFAIK, there isn't direct support for this in IPython, but there are a few gists that get us part of the way there. runipy also looks interesting.

I played around with the gists and runipy, and they work well if python errors are raised. However, if shell commands are run (e.g., !print_qiime_config.py) and there are failures, the cells are reported to have run correctly.

Offhand I haven't found an easy way around this, but there's likely a way to capture the exit status of a command (an ipython session sets _exit_code). This is important to get working since most of our tutorials use shell commands instead of python code.

Other useful links:

http://stackoverflow.com/q/6755794
http://stackoverflow.com/q/9977446
http://ipython.org/ipython-doc/dev/development/messaging.html

[gregcaporaso]

I agree that this is really important. Would doctest help at all? Alternatively, can we set ipython as the python interpreter to let the commands beginning with ! work?

[jairideout]

I tried converting a notebook to python using:

ipython nbconvert --to python foo.ipynb

and then ran the python file using ipython:

ipython foo.py

It successfully runs all commands, but doesn't error out if a shell command fails, e.g. !duck-n-cover. The exit code of the command is 0 too. Need to see if there's a way to get it to fail if _exit_code isn't 0.

Will also look into doctest.

[lkursell]

+1 This idea sounds good, especially for allowing us to write more tutorials in ipynb form, like the one I put together for group_significance.py (that I need to merge after some changes).

On Dec 20, 2013, at 8:15 AM, Jai Ram Rideout notifications@github.com wrote:

I tried converting a notebook to python using:

ipython nbconvert --to python foo.ipynb
and then ran the python file using ipython:

ipython foo.py
It successfully runs all commands, but doesn't error out if a shell command fails, e.g. !duck-n-cover. The exit code of the command is 0 too. Need to see if there's a way to get it to fail if _exit_code isn't 0.

Will also look into doctest.

—
Reply to this email directly or view it on GitHub.