IPython notebooks for astropy tutorials

[cdeil]

It would be nice to have IPython notebooks for astropy tutorial.

This has been mentioned here, but should be discussed independently of that ticket.

Ideally there would be a simple workflow where the rendered notebooks are converted to rst and rendered in the astropy theme.

This is pretty new territory, but here are some references where this or something similar has been discussed / tried:

• http://jakevdp.github.com/mpl_tutorial/#
• https://github.com/ipython/nbconvert/pull/35
• https://github.com/ipython/nbconvert/issues/54
• http://statsmodels.sourceforge.net/devel/examples/index.html

Besides from the technical issue, there's the question of where to put the notebooks and how to integrate it with the existing astropy website or sphinx docs.
Here my suggestion would be to let anyone make astropy notebooks, put them in their own github repos or gists, render them with nbviewer and collect links with one-line descriptions at
https://github.com/astropy/astropy/wiki/notebooks

If that sounds OK I can add that wiki page and make a short astropy intro notebook that demonstrates a few lines of code from each astropy sub-package, probably simply copy & pasting the first example from the sphinx docs of each astropy sub-package. This could be a nice way for people to start exploring astropy.

[cdeil]

I just saw https://github.com/astropy/astropy/wiki/code-examples , which is related.
There's no content there yet.

[embray]

Does nbconvert actually produce rst? I thought at this point it only made HTML, and pdfs to a limited extent.
In any case I think a sphinx extension would have to be written, and if nobody else has done it yet that would be a place to start. If nbconvert can produce rst that's great--otherwise I would just insert the HTML output into HTML output of sphinx. That can be done with Latex too, albeit a bit more trickily.

[jiffyclub]

Yeah it can do rst: https://github.com/ipython/nbconvert It requires pandoc to do most of the actual text conversion, though.

You probably still need a Sphinx extension that handles the directives it uses in the output, but someone may already be working on it.

[eteq]

+1 to the general idea. I wonder if we should look into discussing this directly with the ipython devs (and developing some tools that they will include in ipython)? Or even asking them about developing this with us as the guinea pigs? Ipython just got a grant from the Sloan foundation to support development (http://mail.scipy.org/pipermail/ipython-user/2012-December/011839.html) so now might be a great time to ask them to do new things :)

[cdeil]

Did you guys look at the links in my issue description above?

@jakevdp did write a sphinx notebook extension ( https://github.com/ipython/nbconvert/issues/35 ) and http://jakevdp.github.com/mpl_tutorial/tutorial_pages/tut1.html demonstrates what is already possible.

There is an open issue with some discussion on this topic: https://github.com/ipython/nbconvert/issues/54

[jakevdp]

Hi,
The sphinx extension I wrote is very basic: it essentially uses the nbconvert html output as a raw html input to the document. It makes use of the HTML5 ability to adjust CSS attributes within local divs, so is not compatible with older browsers. A better solution would be great, but I'm not sure how to proceed. We'd love more input to the discussion if you have it: ipython/nbconvert#54

[embray]

@jakevdp Where actually is your Sphinx extension? I didn't notice any links to it in that thread. It might work as a starting point. Though probably a more broadly compatible solution is just to require the inclusion of an extra CSS file in the Sphinx static documents.

[jakevdp]

The sphinx extension is part of nbconvert:
https://github.com/ipython/nbconvert/blob/master/notebook_sphinxext.py

[ellisonbg]

Hi, IPython dev here. Saw the discussion here about developing astropy examples as IPython notebooks. We are actively working on our nbconvert machinery for exporting notebooks to different formats. But there are some issues to be aware of:

• reST has many more features than Markdown. Thus if you write a notebook using Markdown cells, you will never be able to get it to export those fancy reST features you might want.
• You can write a notebook using our raw text cells and actually put raw reST in there. That gives you the full power of reST, but the notebook will be ugly to look at as a notebook.

What this means is that you have to decide what you want to do with the notebooks at the end of the day. Are they going to viewed and worked with as notebooks or will they just be a convenient way of generating sphinx docs. You can make either choice, but try to do both will be difficult.

We do have some ideas on ways we could bring some of the features of reST/Sphinx to Markdown based notebooks. Two of the big ones are:

• Ability for one notebook to link to another in the same directory
• Ability to handle references/bibliography.
• Ability to link to a particular cell in a notebook.

These are going to take some time, but we think they are possible. Because of this, with IPython, we are going to move all of our narrative docs over to Markdown based notebooks. Another point that may be relevant is that nbconvert will likely gain the ability to convert an entire directory tree of notebook to a single latex/pdf file - similar to how Sphinx works. We are also in the process of integrating the default Sphinx latex styles into nbconvert.

We would love to work with the astropy community on these things, but just a warning - nbconvert is very unstable and raw code right now. We expect it to stabilize by mid year. Cheers!

[taldcroft]

@ellisonbg - thanks for the input, that's very useful information. From my perspective the initial focus was limited to generating specific tutorials (analysis threads) as notebooks and then integrate them into our main sphinx docs for viewing as notebooks. Of course there would be links for the user to be able to download and run the notebooks. It would be good be to able to include styles to match (to the extent possible) the look and feel of the sphinx docs.

This brings up a related question that I have about converting existing sphinx tutorials to ipython notebook. If you look at http://python4astronomers.github.com/, there is a mini-course on practical Python for astronomers. We would love to convert this to notebook, updating the examples along the way to use Astropy. For the most part this doesn't use fancy sphinx, just a lot of text and code examples [1]. It seems that with the expected new features in notebook, most of that site could be replicated as notebooks without too much loss.

This leads to my question - are you aware of any tools to convert reST/sphinx to notebook on a best-effort basis, understanding that some manual cleanup will be needed. Or is anything like that planned?

[1] actually there is a little javascript to hide/reveal answers. That sort of notebook functionality is already planned, right?

[ellisonbg]

Tom,

We have thought a little bit about Sphinx to Markdown conversion. One
option would be to run the reST through Pandoc to produce Markdown. Then
it wouldn't be too difficult to run through the Markdown and spit out
notebook cells. I don't think anyone has begun this work yet though.

As far as the hide/reveal of answers, nothing has really done done on this
front. For that, I would expect it to be developed as an extension, not
built into IPython itself though.

On Thu, Feb 21, 2013 at 4:50 AM, Tom Aldcroft notifications@github.comwrote:

@ellisonbg https://github.com/ellisonbg - thanks for the input, that's
very useful information. From my perspective the initial focus was limited
to generating specific tutorials (analysis threads) as notebooks and then
integrate them into our main sphinx docs for viewing as notebooks. Of
course there would be links for the user to be able to download and run the
notebooks. It would be good be to able to include styles to match (to the
extent possible) the look and feel of the sphinx docs.

This brings up a related question that I have about converting existing
sphinx tutorials to ipython notebook. If you look at
http://python4astronomers.github.com/, there is a mini-course on
practical Python for astronomers. We would love to convert this to
notebook, updating the examples along the way to use Astropy. For the most
part this doesn't use fancy sphinx, just a lot of text and code examples
[1]. It seems that with the expected new features in notebook, most of
that site could be replicated as notebooks without too much loss.

This leads to my question - are you aware of any tools to convert
reST/sphinx to notebook on a best-effort basis, understanding that some
manual cleanup will be needed. Or is anything like that planned?

[1] actually there is a little javascript to hide/reveal answers. That
sort of notebook functionality is already planned, right?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/559#issuecomment-13887118.

Brian E. Granger
Cal Poly State University, San Luis Obispo
bgranger@calpoly.edu and ellisonbg@gmail.com

[hamogu]

This issue is going to be addressed through astropy/astropy-tutorials.
Thus, I suggest to close this issue in this repository.