Add a top-level function to execute notebooks

[tritemio]

With all the machinery to run notebooks in-place in nbconvert, would be nice to offer a top-level function to simply execute a notebook programmatically.

I use something like this, if others agree I can send a PR:

def run_notebook(notebook_name):
    """Runs the notebook `notebook_name` (file name with no extension).

    This function executes notebook with name `notebook_name` (no extension)
    and saves the fully executed notebook in a new file appending "-out"
    to the original file name.

    It also displays links to the original and executed notebooks.
    """
    nb_name_full  = notebook_name + '.ipynb'
    display(FileLink(nb_name_full))

    out_path = 'out_notebooks/'
    out_nb_name = out_path + notebook_name + '-out.ipynb'

    nb = nbformat.read(nb_name_full, as_version=4)
    ep = ExecutePreprocessor(timeout = 3600)

    try:
        out = ep.preprocess(nb, {'metadata': {'path': './'}})
    except Exception:
        msg = 'Error executing the notebook "%s".\n\n' % notebook_name
        msg += 'See notebook "%s" for the traceback.' % out_nb_name
        print(msg)
        raise
    finally:
        nbformat.write(nb, out_nb_name)
        display(FileLink(out_nb_name))

Having a function like this would save some gimmick to users wanting to run a series of notebooks in one shot.

[mpacer]

@minrk Is there anything that needs to be done for this issue?

With @tritemio's https://github.com/tritemio/nbrun he seems to have covered the use case, is the idea that we should attempt to integrate that functionality into the core library? What are the barriers to doing that?

One thought is that the approach to distributing nbrun separately is going to discourage people from using it.
Still, it's not clear where it would fit into the nbconvert codebase.

[tritemio]

nbrun also executes notebooks passing parameters, but this is not the topic here. What I was proposing was to add a simple function that given a notebook (i.e. filename) it executes it and writes the executed version to disk.

A use case is executing a set of notebooks on a CI server (for both testing and rendering). Right now, I'm adding nbrun.py to the notebook repository as it is only a small file (and one function). Having such a function directly in nbconvert would avoid the need of vendorizing nbrun.

API-wise, the function should be in the top-level module and have only a few options to specify the output file (including not writing the output, maybe just returning the nb object) and expose a few ExecutePreprocessor traitlets such as kernel_name, timeout and allow_errors.

[mpacer]

Talked with @takluyver about this a bit today. I think there's going to need to be a general discussion of how we want to expose an API for executing notebooks (at least in terms of anything beyond jupyter nbconvert --execute --inplace --to notebook).

@takluyver can correct me but I believe his preferred method would be to have something like jupyter nbexec that by default writes to stdout so that it could be (Bash) piped into (for example) nbconvert and converted (e.g., a use case being where a notebook is being output to html for the purposes of a dashboard).

You suggest above running a series of notebooks in one shot and the way you propose to modify the current functionality sounds like it would be an API expected to work from the module perspective of nbconvert rather than as a CLI. That seems to be a different use case than what @takluyver was thinking of.

One thing that would be useful is a collection of use cases for executable notebooks. Do you know of the uses that people have put executing notebooks in place by using your nbrun code?

[tritemio]

@michaelpacer, I have two use cases. One is testing execution of a set of notebooks (let say a set of library tutorials) both locally or through a CI service. I don't execute the notebook in place but save them in a second file. Another case is reducing code duplication in analysis executing notebooks in batch from another notebooks while passing arguments. Passing arguments to notebook was not my original intent with this proposal. I'm not aware of others using nbrun.

I think that a module-level function that can also be called via command line would work quite well.

[mpacer]

I think that #573 should take care of what you want.