Skip to content

IJulia notebooks for documentation? #4579

Open
StefanKarpinski opened this Issue Oct 19, 2013 · 20 comments
@StefanKarpinski
The Julia Language member

I'm really keen on the idea of writing Julia docs using IJulia notebooks. Notebooks support Markdown text formatting with embedded math equations using MathJax, and of course, if you run a notebook locally, you can execute all the code yourself, which I think will help people learn more effectively and making it easy to step through examples will help keep the examples up-to-date and catch cases where code changes breaks the examples more quickly. I've also found that IJulia notebooks are a great way to write documentation in the first place.

However, it would be nice if the source format, rather than the not-very-human-readable .ipynb JSON format, were something more like Markdown with embedded code examples (and output) that could be transformed – via code execution – to notebook form. This would be a bi-directional transformation:

  • markdown w/ code and output + IJulia kernel ⟹ IJulia notebook
  • IJulia notebook ⟹ markdown w/ code and output

The first transformation occurs via code execution; the second by just outputting the content of the notebook to markdown format, with input and output cells included as embedded code. Or maybe this process is too complicated and .ipynb should just be the source format.

So what are the features that we need from IPython notebook to support this? Obviously, we need the ability to link from one notebook to another. What else?

@StefanKarpinski
The Julia Language member

Relevant IPython issues: ipython/ipython#3806, ipython/ipython#977.

cc: @Carreau, @fperez.

@Carreau
Carreau commented Oct 19, 2013

One of the problem you will have if how to store many output in markdown (json store'em'all), so you will almost never be able to get a round trip .ipynb->md->.ipynb.

We have some kind of importer that will split a python source marked with comment into cells, but it will give you only something partial. that you will probably have to tweak most of the time.

One could use rst with specific sphinx directive to generate a notebook.

.ipynb -> md already kind of exist in nbconvert. It is not perfect as usually a human written document will not have html in it, but the automatic converter have issues to deal with that.

The converter does not add hints for back-conversion to ipynb, it shouldn't be too hard to do though.

@stevengj
The Julia Language member

If there is graphical output (e.g. a plot), then markdown format is not so convenient... I'm not sold on the need for this, as opposed to just using notebook files directly (+ generated HTML/nbviewer output for web access).

@Carreau
Carreau commented Oct 19, 2013

This repo did something similar (IScala Notebook) with a script to import its .xipynb.

@stevengj
The Julia Language member

Here is some info on linking between notebooks.

@StefanKarpinski
The Julia Language member

That seems to be a pretty major roadblock, unfortunately.

@Carreau
@JeffBezanson
The Julia Language member

This would be great for the manual, but the help system is also very important. Am I correct that this would not really address the help system?

@johnmyleswhite
The Julia Language member

I was thinking the same thing: IJulia seems great for making what R calls vignettes (free-form text manuals), but less useful for a help system where you want structured data about functions.

@simonbyrne

Perhaps something analogous to the Mathematica help system: for those who haven't used it, if you look up help from within Mathematica itself, you get something that looks similar to a HTML help page, however you can actually edit and re-run the code parts to see how things work (from what I can tell, it seems to run in its own kernel which is cleared when you leave the page).

A possible vision:
1. Help files are stored in source in some sort of markdown + code for easy storage and editing.
2. During the build process, these are run and stored as ijulia notebooks (this would also function as a type of unit test), and would generate plots etc.
- Static text-based and static html-based versions can also be generated from this.
3. If I call help from:
a. an IJulia console, it opens the IJulia version, while also opening a temporary kernel (or this could be loaded on demand, since most people probably won't want to play around with the examples).
b. a text-based console, it opens the static text-based version.
4. the static html pages could be hosted on the website.

@simonbyrne

On a related note, generating the output at build time (as opposed to saving it in the repo) would help avoid some problems with inconsistent documentation (e.g. #4747).

@pao
The Julia Language member
pao commented Nov 7, 2013

On a related note, generating the output at build time (as opposed to saving it in the repo) would help avoid some problems with inconsistent documentation (e.g. #4747).

Also #2501.

@ViralBShah
The Julia Language member

@shashi has an early converter that converts our documentation to IJulia notebooks.

@shashi
shashi commented Nov 24, 2014

Here https://github.com/shashi/rst2ipynb is the script that can use Pandoc to parse rST and generate valid ipynb files. Has a lot of rough edges, it's possible to fix all of them in a stop-gap manner. The rough edges are there mainly because Pandoc doesn't parse Sphinx variant of the rST.

So a better way of implementing this is as a Sphinx extension (or a Pandoc backend?), which I tried to do till their OOP got to me. Will try again if there is interest.

@MikeInnes
The Julia Language member

Also worth noting that Markdown.jl could handle this kind of thing easily, which might be useful for turning inline documentation into notebooks. Would be kinda cool to type ?fft and have a notebook generated on the fly.

@johnmyleswhite
The Julia Language member

+1 for focusing on Markdown.

@shashi
shashi commented Nov 24, 2014

+1. Definitely a much better direction than sticking to Sphinx.

@staticfloat
The Julia Language member

@one-more-minute I really like the "notebook generated from markdown" approach. Seems like something that fits into the Julia way of doing things very neatly.

@simonbyrne

Possibly relevant to generating ipython notebooks: ipython/ipython#4930

@Carreau
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.