Support jupyter notebooks

[choldgraf]

I've had a few people ask me whether they can simply write their examples in jupyter notebooks, rather than writing them as .py files first. Is there any plan to support things like plot_example.ipynb?

[agramfort]

for me it defeats the purpose/vision of sphinx-gallery

[Cadair]

I wonder if a better idea would be to write a converter from ipynb to a sphinx-gallery .py file. Things like adding in the comment lines around markdown cells etc. Basically the inverse of the .py->ipynb converter in SG itself. This is a workflow I end up doing manually quite a lot, write an example in a notebook, convert to sphinx-gallery format.

[Titan-C]

I think we won't support this. I do understand that many people develop on jupyter notebooks, I did that too for quite some time (cell execution is the main motivation). Nevertheless, jupyter notebooks are a pain to version control(even with nbdime), plain Python files are easy. Second thing an certainly more important is all the magic tricks jupyter notebooks have, and we don't support that either. If people start developing their examples in the notebooks, they'll expect many of the behaviors and features of the notebooks which we don't support, nor plan to support. Some features are a matter of debate(like caching image output #161), because the behavior is not unique between a Python file and a notebook they are not equivalent things. Third, Jupyter uses markdown, Sphinx uses reStructuredText. There is no good parser between them (see #220), we already have a (limited) rst2md, I don't think implementing a md2rst is a good way to go. But people writing on Jupyter notebooks will expect to write in markdown, and it becomes a mental overhead, to keep in mind that you would need to write rst in the notebook for it to be rendered correctly in the Gallery and then transformed to a notebook. Concerning workflow: I have done notebook to python conversion manually quite few times and I feel it works well enough. You can directly, from the notebook export to python, just download as Python file. Then you are left with a python file which has all markdown blocks as code comments like we expect, for a regular python script and what the gallery understands. Most of the job is already done, indeed there is some extra work left to do, like to make your first comment the docstring, to put a line of hashes before each code comment, delete the notebook block index comments, etc. I believe doing this as a conscious step is better than having the converter, where users will expect it to "just work", because a notebook and a code file are "equivalent". After you have your example done, Sphinx-Gallery will always provide you with a notebook version of the code(either download from the build gallery or use the CLI http://sphinx-gallery.readthedocs.io/en/latest/utils.html#convert-python-scripts-into-jupyter-notebooks), which you can edit further and if changes are made you can export again to python file and doing the diff between those is much easier because they are plain text. Finally, if a notebook to python file is really needed. I would certainly accept to make a new repository "sphinx-gallery-contrib", to host it. Like having a hub for community developed utilities and features, which are not supported by the main project.

[GaelVaroquaux]

I wonder if a better idea would be to write a converter from ipynb to a sphinx-gallery .py file.

+1

[choldgraf]

I wonder if a better idea would be to write a converter from ipynb to a sphinx-gallery .py file.

This is a cool idea. I know that the jupyter folks are hoping to improve the nbconvert documentation and make it easier to build new converters etc.

This reminds me of a question that's been nagging at me these last few months: have you all ever thought about splitting off sphinx-gallery into two modules?

• One that does .py -> .rst conversion
• another that does the gallery building etc

?

It seems like the first thing would be useful in-and-of itself, and might get more widespread adoption if it weren't packaged as a sphinx extension (it's basically like RMarkdown which is super popular in the R world). Just a thought!

[Cadair]

@Titan-C I see your point about the limitations of md->rst. I think that it would still be useful if the basic stuff works, even if the user has to manually md -> rst the cells, at least you don't have to do the repetitive things like delete the cell marker comments and add the SG block comments. I am also very happy with the idea that the tool could live in a different package.

I am unlikely to be able to take this on in the near future, but I will keep it in mind unless someone else beats me to it.

[chsasank]

Hey @choldgraf @Cadair, I have already written a tool to convert a ipynb notebook to python file as required by sphinx-gallery. Here it is. It uses pandoc.

It works very well. I have used it to convert old notebooks tutorials to sphinx gallery documentation for pytorch(http://github.com/pytorch/tutorials).

[GaelVaroquaux]

This reminds me of a question that's been nagging at me these last few months: have you all ever thought about splitting off sphinx-gallery into two modules? • One that does .py -> .rst conversion • another that does the gallery building etc

By splitting in modules, do you mean splitting in different projects, or rather giving different and clear access point to facilitate reuse? I think that the second would be great, and we should definitely work on it. I think that the first one would bring in more cost than value, where the cost would be in maintaining two packages, and also having development distributed across them.

[Titan-C]

rather giving different and clear access point to facilitate reuse? I think that the second would be great, and we should definitely work on it.

Yes I have been working on making the codebase more modular, for easier reuse and testing. That was the first aim of closed PR #160, I'm retrying that now on PR #239

[choldgraf]

I think modularizing the code more is a great step! I agree that splitting into multiple modules is something that you'd only wanna do if it was obviously worth it. I think it'd be cool for somebody to point to a single .py file and have it generate the rst / notebooks / whatever from that. E.g. sphinx-gallery myfile.py --to rst

[choldgraf]

as it sounds like people are only so/so on adding jupyter notebooks support, what do folks think about adding a section to the documentation called "Using Sphinx-Gallery with Jupyter Notebook examples" and there suggesting that the best way to do so is to use something like: https://gist.github.com/chsasank/7218ca16f8d022e02a9c0deb94a310fe

We could either link to that gist, or we could add a utility function to SG that people could use in their conf.py file. Would that be a compromise instead of "officially" supporting .ipynb?

[lesteve]

We could either link to that gist, or we could add a utility function to SG that people could use in their conf.py file. Would that be a compromise instead of "officially" supporting .ipynb?

I am fine having a utility function in sphinx-gallery, if "by use in their conf.py file", you mean that it is up to the user to write the code that finds all their notebooks, converts them to .py through our utility function (possibly they need to tweak the docstring with title and first short paragraph), and put them in a folder structure that sphinx-gallery expects (e.g. README.txt).

A few comments:

• tests are needed to make sure this is working
• one way to quickly test the utility function would be to run it on our generated notebooks and quickly browse the diff with our .py example
• I'd be in favour of wording the doc to present this possibility as a manual one-off conversion: basically you convert all of your notebooks and then you can remove your notebooks from your repo. In other words notebooks are not supposed to be a first-class citizen input.
• in the longer term if we have an optional dependency on pandoc we may as well revive Add optional pypandoc rst parsing #220 to do the rst -> md conversion.

[choldgraf]

if "by use in their conf.py file", you mean that it is up to the user to write the code that finds all their notebooks, converts them to .py through our utility function (possibly they need to tweak the docstring with title and first short paragraph), and put them in a folder structure that sphinx-gallery expects (e.g. README.txt).

yep, that's what I was thinking.

one way to quickly test the utility function would be to run it on our generated notebooks and quickly browse the diff with our .py example

Yeah, I'd think that a round-trip from .py -> .ipynb -> .py shouldn't modify the script at all, so that'd be a good test.

I'd be in favour of wording the doc to present this possibility as a manual one-off conversion

I think I'm 👎 on telling people they should or should not use one workflow or another.

I'm still not convinced that a utility function like this is a good idea BTW, curious to hear if this is out-of-scope for the package in some people's opinion. I'm just trying to provide options for the people that are using notebooks in their examples/etc.

[ogrisel]

What about using jupytext python script (with cell boundary markers) to markdown converter + sphinx markdown support via the reconmark extension.

Note: the jupytext python script support works great with VSCode interactive execution window that supports plotting and such.

[ogrisel]

But I am not sure how cross-referencing sphinx sections from the main toc tree would from markdown example would work.

[choldgraf]

I'm +1 on using jupytext for this, it basically duplicates the conversion utility function we've got in SG...that said, IIRC folks were hesitant to add it as a dependency...

[choldgraf]

A note here - the PlasmaPy team has a cool blog post showing how they monkey-patch Sphinx Gallery to generate Binder links that point to .py files, and then use Jupytext to automatically convert these files to ipynb files on-the-fly when the Jupyter server spins up. This would let people create Binder links on sites like ReadTheDocs, which would be pretty cool...

https://stanczakdominik.github.io/posts/simple-binder-usage-with-sphinx-gallery-through-jupytext/

I wonder what folks think about implementing something like this...

[WesleyTheGeolien]

Hey everyone 👋

Just a quick thought on this, I know sphinx-gallery executes the python scripts to generate the ipynb files (which means they are always up to date)

I may have missed something here but rather than going ipynb -> py -> ipynb -> sphinx would it be possible to use nbconvert to convert the jupyter notebook to html directly (then potentially add styling) and insert this into the final gallery behind the link?