Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

add file_to_run=file.ipynb capability to the notebook #1675

Closed
ivanov opened this Issue · 11 comments

8 participants

@ivanov
Owner

over at ipython/nbconvert#13, @Krastanov requested the ability to re-run all cells in a notebook before doing a conversion.
@Krastanov said:

Is there a way to evaluate all cells of a notebook before we use nbconvert?

Why:
I am writing a script that will convert our example notebooks to some static form to be part of the documentation. But many of > the examples contain plots, which take too much space to be kept in git so we are keeping only the unevaluated notebooks in the tree. However we want to generate documentation containing these images, so we must generate them somehow?

I suggested this can legitimately live in the ipython notebook entry point (hence this enhancement request), but can also see the advantages of having that functionality in the (forthcoming) ipython nbconvert

This would be the equivalent of clicking on "Run All" under the "Cell" menu of a notebook and saving it, without needing a web browser.

This is somewhat similar to, but different from #945.

@fperez
Owner

The question is what are the desired semantics: do we want to run the cells and update the nb contents in-place? That's currently more or less impossible, since the code responsible for much of that is actually in the browser. Now, being able to re-execute the code in a notebook by itself, that's a different story. But that would be only to re-generate figures on-disk, for example...

Given how much of the document logic is in JavaScript, I think this is a no-go for the foreseeable future, and I'm not sure it's worth keeping it open since it's likely unrealistic to change even in the long run. @ellisonbg, @minrk, thoughts on this? I'm trying to be fairly aggressive in issue triage so we keep open things we can realistically work on, and I'm not quite sure this falls in that category...

@minrk
Owner

I think writing a Python script that runs code and records output is actually very straightforward. Remember that the output format in the notebook is basically a transcript of the contents of pyout/display_pub messages. The part of the js that records outputs and submits execution is actually extremely basic, and largely duplicated by code in the KernelManager object.

It would basically be:

for cell in nb.worksheets[0]:
    if not code_cell:
        continue
    KM.shell_channel.execute(cell.input)
    code_cell.outputs = []

    reply = KM.shell_channel.get_msg()

    while True:
        try:
            out = KM.sub_channel.get_msg(block=True, timeout=0.1)
        except Empty:
            break
        else:
            if out.msg_type in ('display_pub', 'pyout'):
                code_cell.outputs.append(out)

Note that that was just pseudo-code tossed off the top of my head, but I don't think a real version would differ significantly.

What would be trickier, and probably not a reasonable thing to do, would be to try to fake the message spec with simple exec/run_cell. But using and actual KernelManager should make it pretty simple.

@fperez
Owner

True, it may not be so bad after all... We'd want to preserve all text cells in-place, so it has to be written with a bit of care. But now I see a way to do it, actually; thanks for pointing this out!

@minrk
Owner
@Krastanov

This would be absolutely great for the sympy documentation (the "examples" part that is being ported to notebooks at the moment).

Regrettably, I do not know the internal structure of Ipython very well. If you do not plan to implement this anytime soon (ie in the next few weeks), could you point me to the submodules in question (I really have no idea of your submodule structure, so an example preamble with the needed imports for the @minrk pseudo-code will be greatly appreciated. I should be able to understand the rest by tracing the docstrings)?

@Krastanov

These two scripts by minrk are related:
https://gist.github.com/2620876
https://gist.github.com/2620735

EDIT: added the correct links

@cdeil

I find the scripts mentioned in the last comment very useful to automatically run and test my IPython notebooks.
Would it be possible to include them in the nbconvert repo to make them easier accessible?
They don't do notebook conversion, but if I understand correctly the nbconvert repo is actually the breeding ground for any notebook related tools.

  • Make them executable
  • Maybe combine them in one command line tool with options whether to test the output or exceptions?
@Carreau
Owner

They don't do notebook conversion, but if I understand correctly the nbconvert repo is actually the breeding ground for any notebook related tools.

nbconvert is really axed to converting notebook between format. Once it settle it will be merged with IPython itself.

Feel free to create a repository with thoses scripts, it could perfectly go into a nbutils (for example) repository.

In the end we will certainly need a way to manipulate notebooks in a headless way, so maybe we should think of adding those utility to nbformat itself.

@ellisonbg Though ?

@jseabold jseabold referenced this issue in statsmodels/statsmodels
Closed

tutorial notebooks #689

@jseabold

What has become of BlockingKernelManager from those gists? Is there something else you'd recommend to use instead?

@minrk
Owner

The gists have been updated, but actually need PR #3222 to be merged before they work in master. Not much has changed, just some reorganization and PR #3011, which split the KernelManager into two objects - the KernelManager, which creates / starts / restarts Kernels, and the KernelClient, which communicates with a Kernel via channels.

@paulgb

If anyone is still interested in this functionality, I wrote an external program to run an IPython notebook as a script and optionally export the HTML through nbconvert: https://github.com/paulgb/runipy

@takluyver takluyver closed this in #4452
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.