Fixes #720 : Auto generate ipython notebooks

[quantshah]

Simple Hack to get ipython notebooks out of examples. Needs a little work with markup and equations but other wise should be fine.

@arokem Check sample here

[arokem]

This is definitely going in the right direction. Would it be possible to do this as a separate instruction in the doc Makefile, rather than part of running the examples?

https://github.com/nipy/dipy/blob/master/doc/Makefile

I think it will be cleaner that way.

[quantshah]

@arokem Sorry for the long intervals, is this what you suggested ? I added the code to make ipython notebooks to the dipy/tools/make_examples.py file and as such whenever the valid python examples are built, the notebooks will be built automatically.

I still have some cleaning up to do but, is this a good way to put the notebook conversion code? Or should I have a separate make_notebooks.py file in tools, which is called in a similar fashion as make_examples.py

[Garyfallidis]

@sahmed95 is this stalled for now?

[quantshah]

Need to find the right place to run the code during a build that auto-generates the Ipython notebooks. Also, need to find a way to implement and update gists as @arokem suggested.

Even better, a script that would automatically create/update a gist on github and create the nbviewer page would be fantastic.

Finally, maybe add the links to download the Ipython notebooks in the examples in tutorials.

Will look into it

[arokem]

Would be good to see what sphinx-gallery
http://sphinx-gallery.readthedocs.io/en/latest/ does, and see whether we
can use some of that machinery. We can't really use sphinx gallery as-is,
because the format of our examples is quite different (they only allow one
section of text at the top, and we want to interleave them), but we can see
what they do under the hood for the conversion. We can also roll our own
using nbconvert http://nbconvert.readthedocs.io/en/latest/. Automating
that to happen during the doc build should be rather straight-forward.

On Sun, May 22, 2016 at 9:08 AM, Shahnawaz Ahmed notifications@github.com
wrote:

Need to find the right place to run the code during a build that
auto-generates the Ipython notebooks. Also, need to find a way to implement
and updates gists as @arokem https://github.com/Arokem suggested.

Even better, a script that would automatically create/update a gist on
github and create the nbviewer page would be fantastic.

Finally, maybe add the links to download the Ipython notebooks in the
examples in tutorials.

Will look into it

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#1022 (comment)

[quantshah]

This is languishing as I haven't really worked with Sphinx. I separated the code to generate Ipython notebooks from the standard examples. The function is documented now.

read_example gets the text from a valid example in doc/examples/ and make_notebook converts it into a notebook which can be written using write_notebook

[arokem]

I think it would make sense to integrate a call to this into the Makefile, in a similar manner to the way that things currently work for converting from .py to .rst (https://github.com/nipy/dipy/blob/master/doc/Makefile#L122), so that when the doc html pages are generated, they include a link to the generated notebook file. Does that make sense?

[quantshah]

Yes. Even I initially started towards that and had put the code inside the make_examples.py file. I can put a call there to make_notebooks.py which will build the examples. So where do we put the notebooks (as a separate folder in docs "example_notebooks") I couldn't follow your last suggestion about Sphinx and nbconvert @arokem. Since dipy doesn't use readthedocs.

Do you suggest that we have our own readthedocs and publish examples such as :
http://sphinx-gallery.readthedocs.io/en/latest/auto_examples/index.html

Similarly, http://nbconvert.readthedocs.io/en/latest/nbconvert_library.html#Extracting-Figures-using-the-RST-Exporter also looks great.

I could setup a readthedocs for Dipy following : http://read-the-docs.readthedocs.io/en/latest/getting_started.html#in-rst

I am in favour of nbconvert rather than sphinx-gallery since it also has interleaved text.

[arokem]

No - I don't think we want to set up readthedocs for this. For two reasons: the first is that our documentation will have a hard time building on a free/automated platform like readthedocs, because our examples require large amounts of data and take a long time to run. The second is that @ghoshbishakh is already working on a new version of the website as a GSoC project over here: https://github.com/nipy/dipy_web, so it would be a duplicated effort. For now, all I am suggesting is to automate things so that when we run make html it not only produces the .rst versions of the examples, but also the .ipynb versions. I was pointing to nbconvert because it is a library for converting between different file formats and notebooks, and it might be useful in doing this conversion (instead of the code you wrote here). Might be worth checking it out. Similarly, sphinx_gallery have already built machinery for creating ipython notebooks and integrating them into sphinx documentation, so it might be worth looking what they have done, so we don't need to reinvent the wheel.

[quantshah]

@arokem Thanks for clearing that out. I have the py ->> ipynb conversion script now and added it to the Makefile under "ipynbexamples"

Running "make ipynbexamples" from the /doc folder generates all the valid_examples in the folder "ipython_notebooks". The notebooks aren't executed though and all that this script does is convert the .py files in the "examples" directory to Ipython notebooks. Now, do we convert the notebooks to rst and then add the rst filenames so that make html builds it just like the python examples?

@Garyfallidis Any pointers on where this should go from here ?(Or maybe @ghoshbishakh can take over) Making a ipynb ->> rst conversion seems simple. [http://sphinx-ipynb.readthedocs.io/en/latest/howto.html]

But I 'm not sure where they should be put and how to get them published. A nice way maybe to make an example gallery : http://sphinx-gallery.readthedocs.io/en/latest/auto_examples/just_code.html#sphx-glr-auto-examples-just-code-py
with a download option for getting the notebook or python example.

[quantshah]

Can this be closed now?

[arokem]

Yes. Closing.

[pep8speaks]

Hello @sahmed95, Thank you for updating !

• In the file tools/make_notebooks.py, following are the PEP8 issues :

Line 39:18: W291 trailing whitespace
Line 63:9: E722 do not use bare except'
Line 83:75: W291 trailing whitespace
Line 159:1: E305 expected 2 blank lines after class or function definition, found 1

[arokem]

We should follow up on this one and finish this up!

[dmreagan]

It would be cool to host the notebooks on a service like binder or Colaboratory so beginning users can start playing with things in their web browser. I don't think interactive VTK windows are going to happen, but maybe we could show renders.

[arokem]

That would be cool, but we might also run into the limitations (compute, disk) of these platforms. For the notebooks with VTK, we’ll need to make sure that the notebook is run under xvfb. I think that’s doable.

…

On Tue, Jun 19, 2018 at 5:17 PM David Reagan ***@***.***> wrote: It would be cool to host the notebooks on a service like binder <https://mybinder.org/> or Colaboratory <https://colab.research.google.com/> so beginning users can start playing with things in their web browser. I don't think interactive VTK windows are going to happen, but maybe we could show renders. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#1022 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAHPNl1YANE5iG1y11IA0LoYCG9Y5F_Gks5t-RLsgaJpZM4H5SKx> .

[Garyfallidis]

Good idea! :)

[arokem]

This doesn't currently do anything (as in, the code doesn't run when make ipynbexamples is issued). I will make a separate PR that follows up on this.

[arokem]

This is superseded by #1778