Best practices for documentation of projects

[brownbaerchen]

It becomes increasingly clear to me that good documentation of what you did is just as important as good implementation, since the progress is otherwise lost extremely quickly.
I thought we could have a discussion on how to best document a project.

I started writing in jupyter notebooks because they allow for easy-going combination of text, markdown and images, but they have certain drawbacks. In particular, they are hard to integrate with the CI-pipeline to the point that I gave up on integrating them into the website.
Also, plots I showed there are generated locally on my laptop and not updated when the code is changed.
And another drawback is that git is not great with notebooks. You can display the rendered notebook on gh just fine, but showing diffs is a messy affair to the point that I usually add them to the commit without checking what changed.
Finally, I failed to include python variables in the markdown cells.

In the future, I will, for these reasons, try to keep everything within a README.rst file, which gh also displays nicely.
What kept me from doing this was that I didn't have a good editor for this, but a quick google search directed me to restview, which is a tool that displays the rendered html version of your rst file in a browser and updates automatically every time the rst file is saved.
While writing equations and inserting images in rst is not as nice as in jupyter, the immediate rendering allows for a somewhat smooth experience and comes with the benefit of not having to use jupyter.
Instead, I can use my favourite code editor, which, as it turns out, even supports spell checking.
I am sure that I will struggle to get images displayed on gh a bit, since the paths to the plots are different when executing on my local laptop and when the CI pipeline creates the plots, but I am sure I can find a way to deal with this.

Anyways, I am sure you all have much more experience with these things and I would appreciate some discussion on what are the best tools to use and other advice.

[tlunet]

You can try VSCode : there is many extensions allowing to render both Jupyter Notebook and Markdown file on the side (I'm sure they have something for rst files). Only drawback, you may need one or two days to get use to it (but it's worth it 😉)

About project documentation I believe there is three approaches here :

1. Plain python script that the user has to run.
2. Jupyter Notebook.
3. Readme in markdown (or rst) format.

Not so found of 1 and 3 : python scripts, even well documented, are not very nice to describe theoretical concepts and add equations, which we have plenty of. Markdown or rst is nice for that, but you need to update it by hand each time you change the code (and re-execute on you side the parts of code, copy-paste their output, generate and store the figures somewhere, ...). So not that easy to automatize either.

I do prefer the Jupyter Notebook approach :

• Yes, git diff on them is a bit messy, but since it's text html file, you still can see the differences (see, for instance, here... you just have to ignore the small part that does not correspond to a notebook modification, but more to the notebook re-run).
• Yes, you have to re-run and save them each time you modify a part of it (that's maybe a reason why the figures were changed only locally on your computer ... save is not automatic in jupyter notebook once you re-run a cell, but Ctrl + S does the trick). There is also a way to automatize the full re-run of Jupyter Notebook, see here ..., that could be used in a utility script (that re-run all jupyter notebooks in the repo).
• Some python projects achieve to include Notebook in there documentation, see for instance with the Dedalus Project : they have tutorial notebooks, that are rendered in their generated documentation, even using some additional images in the notebook. Imagine pySDC documentation with that 🤓 !
• Notebook stores both the output and the generated plot in image format, so you don't need to store them somewhere near in your repository. It's an all-in-one approach that makes each Jupyter Notebook fully independant (except of course, if you include supplementary non-generated images in your notebook).

In general, I would suggest :

• writing base (theoretical) documentation, that is not code-dependant, using markdown or rst files (not so sure about keeping going with rst file tho 😏)
• writting code-based documentation (tutorials, project description, results, ...) using Jupyter Notebook
• add a re-run of all notebook in the CI, that should run without error

Eventually, there could be some specific approach for project and examples using parallelization (in space and/or time) ...

[brownbaerchen]

Well, the website is written from rst files and the images are automatically integrated there after they are created. You can do the same with text, I guess. So you write a python script that writes an rst file with some output of the script, which you can include from another rst file. Yes, it's a little bit messy, but you can automise this and make it pretty by including rst styling in the string that you print to file.

I spent about a day on trying to do this with notebooks, but it was a bit tricky to automise their execution.
I managed that with a file where you had to put in the path to your notebook, but then I failed to have sphinx look for the notebooks outside of the docs directory to put them on the website.
Yes, you can do all of it, but I would prefer to stick with the tests as they are now and not have a separate pipeline which executes the notebooks. Also, I am not sure how well they work with pytest.

One more thing I don't like about the notebooks is that you need to call some python code to get plots in there. I learned from you to write a script somewhere else so I only need to call a function from the notebook, but it still interrupts the file a little bit.

I guess, I will try to get my hands dirty with the rst files and maybe after gaining some experience with them I will agree with you that notebooks are best.

Thanks for the input!

[tlunet]

Don't hesitate to look at what others projects are doing (cf Dedalus) : you don't need to reinvent the wheel, and usually there is always one person in the world that have found an optimal solution to do what you want ;)

[tlunet]

PS : "write a script somewhere else so I only need to call a function from the notebook" => this should be added in some "utility" subpackage of pySDC, and be the most generic as possible, so other (user or notebook) can eventually use it too

[tlunet]

https://docs.readthedocs.io/en/stable/guides/jupyter.html

Seems like a pretty neat alternative to current pySDC tutorial format ... ?

[brownbaerchen]

It would improve them as tutorials quite a bit, yes. But I once tried using notebooks in sphinx and I failed.
I am sure that other people can absolutely do this, but I found two problems:

• I could not add notebooks from outside the website source directory, i.e. in some project folder
• The notebooks need some special care to be executed by the CI pipeline

Again, both these problems can be overcome by someone who knows what they're doing, but it may require a bit of patience :D