nbconvert: Remove IPython magic commands from --format="python" output

[dpsanders]

nbconvert with the python output format:
ipython nbconvert --format="python"

currently leaves untouched the IPython magics, which get output to the .py.
This is a bug, since the output is no longer a valid Python script.

Indeed, even trying to run the output from the command line with ipython chokes on the magic commands (which is presumably another, separate, bug?)

Summary: the magics must be stripped out.

[takluyver]

It's quite easy to write code that won't work if you just strip out IPython special syntax - e.g. use files = !ls and then refer to files.

I think we should save the transformed code, which will always be syntactically valid Python, but will need to be run in IPython if you've used any IPython special features.

And no, it's not a bug that ipython chokes on magic commands in a .py file, because it expects those to be valid Python. It can handle magic commands in .ipy files.

[minrk]

@takluyver is right - they should not just be stripped. We should do one of two things (or both, possibly):

1. call the output "IPython", since it is a valid IPython script, and give it the extension .ipy
2. apply our input transforms, so that it is indeed valid Python - it still may not work in the absence of IPython,
   depending on what is used, but it would at least be a Python script.

[dpsanders]

The .ipy solution seems like a good solution.
Then we can provide a simple script to remove the magics if necessary:

remove_magic script contains:

awk 'substr($0,1,1) != "%"' $1 > $1

Isn't awk amazing :)

Oh, have to remove the ! stuff too; I guess that's harder and does need transformers.

[dpsanders]

So we need two separate formats; I propose --format=ipy and --format=py.

[minrk]

that awk solution isn't actually general - you will learn that input transforms are much more complicated than that.

[takluyver]

I'd leave the name --format=python for the transformed commands.

We could add an 'ipy' output format, but I don't think .ipy files are widely used, and we've never particularly promoted them. It's also not completely trivial, because we'd need to take blank lines out of things like cell magics, so that it could work out where they finished. In some cases, removing blank lines could change the effect of a cell magic.

[dpsanders]

OK, it certainly sounds like a can of worms.

But the main point to fix right now is that we claim, implicitly or
explicitly, that the output is a pure Python script which can be executed
with python, whereas actually both python and ipython choke on the
output if there are magics (which there will be in a large percentage of
use cases, due to %matplotlib). We are actually outputting an unusable
script.

Since, on the other hand, it suddenly becomes usable if the extension is
changed to .ipy and run with ipython, perhaps now is a good moment to
start promoting this idiom. In this case, %matplotlib inline should
automatically have the inline removed.

On Sun, Jul 21, 2013 at 3:45 PM, Thomas Kluyver notifications@github.comwrote:

I'd leave the name --format=python for the transformed commands.

We could add an 'ipy' output format, but I don't think .ipy files are
widely used, and we've never particularly promoted them. It's also not
completely trivial, because we'd need to take blank lines out of things
like cell magics, so that it could work out where they finished. In some
cases, removing blank lines could change the effect of a cell magic.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3707#issuecomment-21316700
.

Dr. David P. Sanders

Profesor Titular "A" / Associate Professor
Departamento de Física, Facultad de Ciencias
Universidad Nacional Autónoma de México (UNAM)

dpsanders@gmail.com
http://sistemas.fciencias.unam.mx/~dsanders

Cubículo / office: #414, 4o. piso del Depto. de Física

Tel.: +52 55 5622 4965

[Carreau]

But the main point to fix right now is that we claim, implicitly or
explicitly, that the output is a pure Python script which can be executed
with python, whereas actually both python and ipython choke on the
output if there are magics (which there will be in a large percentage of
use cases, due to %matplotlib). We are actually outputting an unusable
script.

That the reason why nbconvert is in tech preview, there are many things at which nbconvert will not be good at, and that we will need to change. The export to .py (even without nbconvert) has always had quirks, but with the nice nbconvert api it should be easier to fix.

I guess be can have a ipy2py cell filter it will help, but I don't think promoting .ipy files is a good idea.
They are convenient but will never replace pure python script as the same way translated magic are just
a nice hack but shouldn't be used in production.

[takluyver]

Also, one of the main reasons to export a .py file is so you can use it as
an importable module. That doesn't work for .ipy files.

Further, I don't know of any editor that will recognise a .ipy extension,
so you lose file associations, syntax highlighting etc. by saving it as
that.

On 22 July 2013 05:20, Matthias Bussonnier notifications@github.com wrote:

But the main point to fix right now is that we claim, implicitly or
explicitly, that the output is a pure Python script which can be executed
with python, whereas actually both python and ipython choke on the
output if there are magics (which there will be in a large percentage of
use cases, due to %matplotlib). We are actually outputting an unusable
script.

That the reason why nbconvert is in tech preview, there are many
things at which nbconvert will not be good at, and that we will need to
change. The export to .py (even without nbconvert) has always had quirks,
but with the nice nbconvert api it should be easier to fix.

I guess be can have a ipy2py cell filter it will help, but I don't think
promoting .ipy files is a good idea.
They are convenient but will never replace pure python script as the same
way translated magic are just
a nice hack but shouldn't be used in production.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3707#issuecomment-21324129
.

[damianavila]

I am biased towards the filter/transform solution (ipy2py)... I do not think it would be a good idea to follow the path of a new .ipy format file, essentially, because all the things @takluyver has pointed out.

[minrk]

We do now perform input transforms in the Python template.

[jclosure]

There are still a few wrinkles that can crop up from the literal input transforms of certain magics, i.e. %pylab notebook or %matplotlib notebook. Since I use the notebook backend in jupyter notebooks, when these are converted with --to python, the output gets converted to:

get_ipython().magic(u'pylab notebook')

This is problematic in ipython and python, because the notebook backend is only available in jupyter. It's possible to modify this pattern in postprocessing to something like:

get_ipython().magic(u'pylab auto')

This works in ipython, but get_ipython() is only available in ipython, so it won't work in python.

Since I want the converted .py scripts to be runnable in python, the best option was to create a filter and a template to comment out these magics all together.

Filter:

import re

# This is a list of magics that we do not want in our output scripts
magics_regexs = [
    r"get_ipython\(\).magic\(u\'pylab .*\'\)",
    r"get_ipython\(\).magic\(u\'matplotlib .*\'\)"
]

def comment_magics(input, **kwargs):
    lines = input.splitlines(True)
    output = ""
    for line in lines:
        new_line = ""
        for regex in magics_regexs:
            if re.match(regex, line):
                new_line = new_line + "#" + line
        if new_line:
            output = output + new_line
        else:
            output = output + line
    return output

Template:

{% extends 'python.tpl'%}
{% block input %}
  {{ super() | comment_magics }}
{% endblock input %}

[NicWayand]

@jclosure Can you provide an example of how to apply that filter to a .py file? (sorry not familiar with templates). Thanks!

[shett044]

Found a better way @dpsanders and @nickweseman :

• Create a template file named "simplepython.tpl". Copy the below statements.

 {% extends 'python.tpl'%}
## Comments magic statement
 {% block codecell %}
 {{  super().replace('get_ipython','#get_ipython') if "get_ipython" in super() else super() }}
 {% endblock codecell %}

• Save simplepython.tpl.
• Type in command line:

jupyter nbconvert --to python 'IPY Notebook' --template=simplepython.tpl --stdout

[NicWayand]

FWIW I wrapped @jclosure's filter to a command line executable https://github.com/NicWayand/hacks/blob/master/pyconvert.sh

[rdbisme]

Just for future, if you want to use @jclosure filter, create a nbconvert_config.py file like this:

import re

c = get_config()

# This is a list of magics that we do not want in our output scripts
magics_regexs = [r"get_ipython\(\).*"]


def comment_magics(input, **kwargs):
    lines = input.splitlines(True)
    output = ""
    for line in lines:
        new_line = ""
        for regex in magics_regexs:
            if re.match(regex, line):
                new_line = new_line + "#" + line
        if new_line:
            output = output + new_line
        else:
            output = output + line
    return output


# Export all the notebooks in the current directory to the sphinx_howto format.
c.NbConvertApp.notebooks = ["*.ipynb"]
c.Exporter.filters = {"comment_magics": comment_magics}

# https://github.com/ipython/ipython/issues/3707/#issuecomment-327971946

And a template file python_nomagic.tpl like this:

{% extends 'python.tpl'%}
{% block input %}
  {{ super() | comment_magics }}
{% endblock input %}

Then you can do:

jupyter nbconvert --config <path/to/nbconvert_config.py> --to python --template <path/to/python_nomagic.tpl> <file.ipynb>

[sappelhoff]

Does somebody have a solution that's as nice and simple as @shett044's in #3707 for nbconvert > 6.0?

They have overhauled the way templates work, and the previous way does not work anymore. See also https://blog.jupyter.org/the-templating-system-of-nbconvert-6-47ea781eacd2

[vincentqb]

See for new format: jupyter/nbconvert#1428 (comment)

[vincentqb]

Leaving a note here to adjust the solution above for cell magic (e.g. %%time), with nbconvert > 6.0.

In nbconvert_config.py:

def comment_magics(input, **kwargs):
    if input.startswith("%"):
        input = "# " + input
    return input


# Export all the notebooks in the current directory to the sphinx_howto format
c = get_config()
c.NbConvertApp.notebooks = ["*.ipynb"]
c.Exporter.filters = {"comment_magics": comment_magics}

In python_nomagic/index.py.j2:

{%- extends 'null.j2' -%}

## set to python3
{%- block header -%}
#!/usr/bin/env python3
# coding: utf-8
{% endblock header %}

## remove cell counts entirely
{% block in_prompt %}
{% if resources.global_content_filter.include_input_prompt -%}
{% endif %}
{% endblock in_prompt %}

## remove markdown cells entirely
{% block markdowncell %}
{% endblock markdowncell %}

{% block input %}
{{ cell.source | comment_magics | ipython2python }}    
{% endblock input %}

In python_nomagic/conf.json:

{
    "base_template": "base",
    "mimetypes": {
        "text/x-python": true
    }
}

Convert with:

jupyter nbconvert --config ./nbconvert_config.py --to python --template ./python_nomagic notebook.ipynb