DOC: Add section on executing notebooks

[tritemio]

This section shows how to execute notebooks using nbconvert API. The executed notebook is saved in notebook format. There is no discussion on exporting notebooks to other format (topic already addressed in another section of the docs).

This is heavily based on this blog post: Batch Execution of Jupyter Notebooks.

Except for links/formatting fixes it should be ready for review. I can rebase to clean history before merge.

[takluyver]

It's probably better to refer to this by name, because if we rearrange the docs or put another page in between, we won't think about this.

[tritemio]

@takluyver, thanks for the comments. I fixed the the points you raised.

Still, I left a note on kernel_name and regarding the command-line example (--execute flag), it is already there.

[tritemio]

Wait for a few fixes left.

[tritemio]

Done with fixes.

• Locally, it builds with no warnings and the internal links work.
• Added example of command-line usage in the section intro
• Improved docstring for timeout and allow_errors traitlets
• Document ExecutePreprocessors.process and add it to API docs
• Reworded docs index intro

Would be nice to add the ExecutePreprocessor traitlet docstrings to the API docs
but I don't know if it's possible using autodoc.

[tritemio]

I also rebased to clean the history.

[willingc]

@tritemio Thanks for this PR. It's very helpful. I have some grammar and style edits to make the section more user friendly. 😎

[willingc]

Python

[takluyver]

@willingc is right, it should be a capital P here.

[tritemio]

@willingc I applied your fixes. Let me know if you want to further tweak the intro.

[tritemio]

@Carreau, I think your example on exporting notebooks should go on a different section, for example here:

• Python API for working with nbconvert

However this section is currently a mess and should be heavily re-edited IMHO.

@willingc, @Carreau, @takluyver is there a way to make traitlets descriptions appear in sphinx API documentation?

[takluyver]

is there a way to make traitlets descriptions appear in sphinx API documentation?

I don't think so, unfortunately. Maybe it could be done with a Sphinx extension, if somebody is interested in writing one.

[tritemio]

@takluyver, regarding the with statement, on a second though, I think in this case it may be ok not to use it. Take this example (reading):

nb = nbformat.read(open(notebook_filename), as_version=4)

No reference to the opened file is saved and the file is closed when nbformat.read() returns. The same goes for writing:

nbformat.write(nb, open('executed_notebook.ipynb', 'wt'))

And, in the final example on error handling, adding the with inside the finally becomes a little heavier on the eyes:

    try:
        out = ep.preprocess(nb, {'metadata': {'path': run_path}})
    except CellExecutionError:
        msg = 'Error executing the notebook "%s".\n\n' % notebook_filename
        msg += 'See notebook "%s" for the traceback.' % notebook_filename_out
        print(msg)
        raise
    finally:
        with open(notebook_filename_out, mode='wt') as f
            nbformat.write(nb, f)

What's your take on this?

[takluyver]

On Python 3, a file that isn't explicitly closed and gets cleaned up when all references to it are gone generates a ResourceWarning. Admittedly these aren't visible by default, and perhaps it's overly strict, but they do show up during tests, so I still think it's better to point people to with open(....

[tritemio]

Ok, I didn't know that, I'll leave the PR as is then, using with open(....

[tritemio]

@takluyver

ResourceWarning is ignored unless Python was built in debug mode.

https://docs.python.org/3.5/library/warnings.html#default-warning-filters

[takluyver]

Yep, like I said, they're not visible by default. You can enable them without rebuilding Python, though, and doing that when testing is good practice.

[Carreau]

seeing that this is an improvement on our docs I would be +1 to merge and iterate.

[tritemio]

Regarding the verbosity of loading, executing and saving the notebook, I proposed in the past a top-level function #125 to run notebooks in one step.

Nowadays I use a slightly improved version of that function which allows to optionally pass parameters to the notebook to be executed (~like a function). See nbrun, rudimentary but incredibly useful in my workflow.

[willingc]

@Carreau +1 on merge and iterate. Will leave to you or @takluyver to merge.

@tritemio Thank you! 🍪

[tritemio]

@willingc, thanks, glad I earn a 🍪!

[willingc]

@tritemio 👍 Also, 🏄 🌴 🍰 is a virtual vacation ;)

[tritemio]

Just added one last commit, I'm done now.

[takluyver]

Looking at the code, it actually raises TimeoutError on Python 3 and RuntimeError on Python 2 (because TimeoutError is new in Python 3)

[tritemio]

@takluyver fixed in the last commit.

[takluyver]

Thanks for all your work on this!

[willingc]

Yes! Thanks @tritemio for the good work and @takluyver for the helpful reviews.

[Carreau]

🍰

[tritemio]

👍 @willingc @Carreau @takluyver