ReST support in notebooks

[minrk]

This is a place to discuss the requirements of ReST support in the notebooks.

We use markdown now, because what we have are light text cells, for annotating your documents. They are essentially rich text + math comment areas. ReST is a great deal more than that, and as such also a great deal more expensive to use (markdown does not require any server communication at all). We initially supported, but removed cell-level ReST support, because it isn't really ReST if each cell is treated as an independent document.

Users should be able to get everything they would expect from ReST when using it in the notebook. That means citations, references, links, and section-levels must be preserved across all cells in a document. As I understand it, this means that all cells must be rendered together. The potential cost of rendering the cells together may mean that render should be a deliberate action, rather than passive on mouseout of every text cell. Further, if ReST has to render the entire document, maybe it should include the code/output as well, and generate the entire document. If we do this, it should probably load in an external preview window, showing the full rendered output, rather than rendering inplace like the markdown cells.

Having markdown as the first-class language for cells, and ReST for full-document output may cause some problems, but it's possible a one-time pass of pandoc for converting md cells to rst would ameliorate that.

[ellisonbg]

I like your idea to have the ReST preview open a new tab/window and it should be a deliberate action on the part of the user.

[fperez]

Here are my notes about reST support from a recent email discussion (it may repeat some of Min's points, I'm just pasting my whole email for reference). None of this is cast in stone, just putting it out here so we have material for the discussion:

1. There's one nasty problem with reST in that its specification of heading levels is ambiguous and resolved at runtime by looking at the whole document (i.e., ==== doesn't uniquely specify the heading level until the other underline types have been looked at). To solve this problem, we're going to take a slightly different approach: users
   will NOT type headings directly into text cells, but instead we will have a 'heading' type of cell, with a level (say 0-9). These cells will determine the document structure, letting us:

• produce output reST that is consistent in its use of heading underlines.
• offer (eventually) document navigation, level folding and structural operations such as moving, promoting and demoting entire sections.

1. Once structural cells are in place, we'll also have reST text cells (in addition to the current markdown ones). These can initially be left completely unrendered, though eventually it would be nice to be able to render them at least cosmetically (i.e. only resolving cell-local markup and links but not trying to render the whole document).
2. There would be a button/menu action to force a server-side render of all the reST in a notebook. This would take longer, but would give fully working cross-references, etc.
3. Given a .ipynb, it should be possible to export it to reST via the command line. This would make it possible to have for example a make rule like:

%.rst : %.ipynb
       ipython notebook export-rest $(NBFLAGS) $<

You could then imagine having a complex sphinx document (even a book) where certain parts are hand-written reST files and other sections/chapters are actually written as notebooks. The flags would allow you to control whether the notebook is only exported from the file as-is, or actually re-executed before export (allowing for a much more sophisticated form of tutorial/documentation creation for libraries, as well as a doctest-like capability of a higher degree).

The reST export will require a fair amount of machinery: currently in the notebook all data is inlined (figures, etc) and that will have to go somewhere. I think a sensible solution is to add a directory called _files/ (like browsers do when they download a full html page), where figures and other files get disentangled and saved. The export will then create .. image:: references to these.

1. I think our focus should be on high-quality Sphinx export, and we leave latex/pdf/html generation in sphinx's hands. We may need to produce a few sphinx extensions (such as improving Ondej Certik's dollarmath one for writing TeX like this: $x+1+\gamma$), but that's it. Beyond that, shpinx does a good job and we should use it.

We talked about Sphinx at EuroSciPy. What are your plans regarding
back-and-forth conversion between json and restructuredtext ?

I don't think we should focus too much on lossless roundtrip conversion: ensuring perfect roundtripping is devilishly hard and making such a promise to users is a hard to maintain contract. However, we could certainly implement reST import so that from a reST document, code blocks are read in as cells. That would make it possible for someone to migrate a reST document into a notebook. Basically this would mean writing a docutils writer that outputs ipynb files.

[ellisonbg]

Implemented as Plaintext cells in master.