Add preprocessor to execute notebooks #5639

jvns · 2014-04-16T15:13:38Z

This adds a preprocessor to execute all the code cells in a notebook.

I basically copied all of this code from @damianavila's gist here: https://gist.github.com/damianavila/5208296.

I don't totally understand all of the error handling. This probably wants more robust unit tests, but I'm not sure exactly what I should be testing.

jvns · 2014-04-16T15:16:47Z

I also shut down the kernel in __del__, and there may be a better place.

minrk · 2014-04-16T19:22:00Z

IPython/nbconvert/preprocessors/execute.py

+        self.km = KernelManager()
+        # run %pylab inline, because some notebooks assume this
+        # even though they shouldn't
+        self.km.start_kernel(extra_arguments=['--pylab=inline'], stderr=open(os.devnull, 'w'))


it definitely should not start with --pylab=inline

+1 for making it an option; it should handle --matplotlib=inline as well

No, it should handle neither.

My mistake, I somehow misinterpreted your comment.

I think it should be an option, defaulting to no arguments. As much as we might like everyone to never run the notebook with --pylab inline, people do run it with arguments.

Issues is in 3.0, starting notebook server with --pylab=inline will not work. So kernel started with --pylab=inline in nbconvert will make no sens. The only thing that will make sens is chosing the kernel you want to use to run the notebook.

It should not handle the --pylab option... I agree...

That makes sense. What would be the best way to make it possible to run it with %pylab inline or %matplotlib inline that would work in 3.0?

paulgb · 2014-04-16T20:28:34Z

Thanks @Carreau for pointing me here.

@jvns feel free to test with the runipy test suite. It's small but covers basically everything that was broken in the original runipy and later fixed: https://github.com/paulgb/runipy/tree/master/tests

I'm happy to see this behaviour making its way into IPython!

Carreau · 2014-04-16T20:54:51Z

hum, Travis seem unhappy because of pandoc. restarting it.

Carreau · 2014-04-17T15:29:20Z

Travis still seem to get the wrong pandoc... wondering why... restarting again.

jlstevens · 2014-04-21T13:14:26Z

Hi

I'm very glad to see this functionality making its way into IPython. I would just like to ask whether this design will allow me to batch run notebooks from the commandline?

So far I have not found a decent way to automatically run notebooks so that display hooks are run properly and that the resulting notebook is saved back to disk (with the fresh display output). This is very important to me as a Topographica developer as that project is now making heavy use of notebooks and we need this type of automation for testing purposes.

My unsatisfactory solution has been to hack my own copy of runipy (which was a very helpful starting point) to load our custom IPython profile, run a %load_ext command (with shell.execute) and then save the result on completion to a new .ipynb file.

I am eager to see that this execute preprocessor can help me to avoid such hackery in IPython 3.0!

Thank you

Carreau · 2014-04-21T14:58:15Z

I'm very glad to see this functionality making its way into IPython. I would just like to ask whether this design will allow me to batch run notebooks from the commandline?

I have to check, but IIRC not yet, there are still a few missing pieces in nbconvert. In 3.0 I would say probably.

jdfreder · 2014-04-21T21:28:04Z

I have to check, but IIRC not yet, there are still a few missing pieces in nbconvert. In 3.0 I would say probably.

#5691

jvns · 2014-04-24T22:18:32Z

Okay, I added the tests from runipy (thanks @paulgb!), and fixed the issues @minrk pointed out (thanks!).

Are there more issues blocking this? I'd like to make it possible in some way to run notebooks with %pylab inline and %matplotlib inline, but that could be a separate PR after this is merged, since there seems to be disagreement about that.

jvns · 2014-04-24T22:19:22Z

The Travis build seems to be failing but I think that's because of pandoc.

minrk · 2014-04-24T22:21:15Z

I'd like to make it possible in some way to run notebooks with %pylab inline and %matplotlib inline

This should already work as long as the %matplotlib magic is called in the notebook itself. If that is not in the notebook, then the notebook shouldn't be expecting inline figures.

minrk · 2014-04-24T22:25:34Z

The tests are failing because your test files aren't installed. I think you will need to add a glob to setupbase.find_package_data.

jvns · 2014-04-24T22:27:11Z

what does "your test files aren't installed" mean?

jvns · 2014-04-24T22:29:06Z

so I agree that it shouldn't be expecting inline figures, but in real life lots of people have notebooks that they've created with --pylab inline or --matplotlib inline. It seems like it would be more useful to people if it could handle legacy things (and maybe give a warning).

minrk · 2014-04-24T22:30:26Z

Sorry, files that don't end with '.py' in the source tree are not installed by default. You have to tell Python to install things things via a structure called 'package_data', which is populated in setupbase.find_package_data. You will need to add file globs for the new test files you have added here, in order for them to be included in an install of IPython.

jvns · 2014-04-24T22:39:33Z

I changed find_package_data(). We'll see if it fixes Travis! Thanks @minrk =)

minrk · 2014-04-24T23:45:28Z

IPython/nbconvert/preprocessors/execute.py

+import os
+import sys
+
+from Queue import Empty


This is queue on Python 3.

jhamrick · 2014-05-07T06:49:59Z

I'm really excited to see this becoming a preprocessor! 👍

I was playing around with it a bit, and found that the prompt number actually isn't being updated in the notebook. I think rather than checking if the message type is pyout and then setting the prompt number only for the output, you need to check if execution_count is in the content and then set the prompt number for both the cell and the output.

Happy to submit a pull request to your branch @jvns if you'd like!

jvns · 2014-05-07T12:22:17Z

yes please!
On May 7, 2014 2:50 AM, "Jessica B. Hamrick" notifications@github.com
wrote:

I'm really excited to see this becoming a preprocessor! [image: 👍]

I was playing around with it a bit, and found that the prompt number
actually isn't being updated in the notebook. I think rather than checking
if the message type is pyout and then setting the prompt number only for
the output, you need to check if execution_count is in the content and
then set the prompt number for both the cell and the output.

Happy to submit a pull request to your branch @jvnshttps://github.com/jvnsif you'd like!

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/5639#issuecomment-42394866
.

Set input/output cell prompt numbers

jlstevens · 2014-05-14T15:14:10Z

Just to say I've just tested out this PR and it largely works as I hoped. Unfortunately, I notice that sometimes the ordering of output is wrong with output appearing underneath an unrelated code cell.

Otherwise, I'm looking forward to using this as soon as it is merged!

minrk · 2014-05-14T20:37:54Z

@jvns can you rebase the PR? There are going to need to be a few changes due to recent things in the message spec, but the PR needs to be based on the latest master, first. If not, I can do the rebase.

minrk · 2014-05-14T20:38:15Z

Sorry, that was for @jvns.

jvns · 2014-05-16T02:18:23Z

I'm a bit swamped right now and probably won't get to this for a bit =\

minrk · 2014-06-26T20:24:54Z

I've rebased this and added the necessary changes for msg spec 5 in #6053.

minrk · 2014-07-01T16:49:34Z

Merged as #6053. Thanks!

Add execute preprocessor closes ipython#5639

Add preprocessor to execute notebooks

fe357db

s/clearoutput/execute/

b842148

minrk reviewed Apr 16, 2014
View reviewed changes

jvns and others added 3 commits April 17, 2014 15:21

Add execute preprocessor to whatsnew

1740d5b

Remove extraneous whitespace

f7d217f

Do self.log.error instead print >> sys.stderr

4b70c36

julia-stripe added 7 commits April 24, 2014 10:04

Move kernel starting / stopping into preprocess()

3215531

Fix merge conflict

41b8fbc

Fix broken create/shutdown code

2ca770d

Replace execute(pass) with kernel_info()

8abe5d0

Start adding tests from runipy

9baedb9

Fix bug with SVG checking

3975950

Refactor assert_notebooks_equal

c85b0b9

Oops, preprocessor*s*

7dda4bf

minrk reviewed Apr 24, 2014
View reviewed changes

IPython/nbconvert/preprocessors/execute.py

import os

import sys

from Queue import Empty

Copy link

Member

minrk Apr 24, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is queue on Python 3.

julia-stripe added 3 commits April 24, 2014 23:04

Fix queue import for Python 3

f0015a0

map returns an iterator in Python 3

f0585c8

print is a function in Python 3

cca0bfb

Carreau mentioned this pull request Apr 29, 2014

runipy extension as core functionality #5741

Closed

minrk added this to the 3.0 milestone May 7, 2014

Carreau mentioned this pull request May 8, 2014

Followup from twitter iiSeymour/Flasked-Notebooks#1

Open

jhamrick and others added 2 commits May 8, 2014 12:24

Properly set prompt numbers

a6e1162

Merge pull request #1 from jhamrick/prompt-numbers

5b7fc38

Set input/output cell prompt numbers

tbekolay mentioned this pull request May 19, 2014

Clean up notebooks nengo/nengo#171

Closed

10 tasks

takluyver assigned minrk Jun 26, 2014

minrk mentioned this pull request Jun 26, 2014

Execute preprocessor #6053

Merged

minrk closed this in f64a9ca Jul 1, 2014

minrk modified the milestones: no action, 3.0 Jul 1, 2014

mattvonrocketstein pushed a commit to mattvonrocketstein/ipython that referenced this pull request Nov 3, 2014

Merge pull request ipython#6053 from minrk/execute-preprocessor

4ea48c1

Add execute preprocessor closes ipython#5639

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add preprocessor to execute notebooks #5639

Add preprocessor to execute notebooks #5639

jvns commented Apr 16, 2014

jvns commented Apr 16, 2014

minrk Apr 16, 2014

paulgb Apr 16, 2014

minrk Apr 16, 2014

paulgb Apr 16, 2014

jvns Apr 17, 2014

Carreau Apr 17, 2014

damianavila Apr 17, 2014

jvns Apr 24, 2014

paulgb commented Apr 16, 2014

Carreau commented Apr 16, 2014

Carreau commented Apr 17, 2014

jlstevens commented Apr 21, 2014

Carreau commented Apr 21, 2014

jdfreder commented Apr 21, 2014

jvns commented Apr 24, 2014

jvns commented Apr 24, 2014

minrk commented Apr 24, 2014

minrk commented Apr 24, 2014

jvns commented Apr 24, 2014

jvns commented Apr 24, 2014

minrk commented Apr 24, 2014

jvns commented Apr 24, 2014

minrk Apr 24, 2014

jhamrick commented May 7, 2014

jvns commented May 7, 2014

jlstevens commented May 14, 2014

minrk commented May 14, 2014

minrk commented May 14, 2014

jvns commented May 16, 2014

minrk commented Jun 26, 2014

minrk commented Jul 1, 2014

Add preprocessor to execute notebooks #5639

Add preprocessor to execute notebooks #5639

Conversation

jvns commented Apr 16, 2014

jvns commented Apr 16, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paulgb commented Apr 16, 2014

Carreau commented Apr 16, 2014

Carreau commented Apr 17, 2014

jlstevens commented Apr 21, 2014

Carreau commented Apr 21, 2014

jdfreder commented Apr 21, 2014

jvns commented Apr 24, 2014

jvns commented Apr 24, 2014

minrk commented Apr 24, 2014

minrk commented Apr 24, 2014

jvns commented Apr 24, 2014

jvns commented Apr 24, 2014

minrk commented Apr 24, 2014

jvns commented Apr 24, 2014

Choose a reason for hiding this comment

jhamrick commented May 7, 2014

jvns commented May 7, 2014

jlstevens commented May 14, 2014

minrk commented May 14, 2014

minrk commented May 14, 2014

jvns commented May 16, 2014

minrk commented Jun 26, 2014

minrk commented Jul 1, 2014