Major overhaul of documentation

[teutoburg]

No description provided.

[hugobuddel]

To understand better where this PR is going, how should we/users interact with these markdown based notebooks? I can't just open them in jupyter notebook:

I get that we can jupytext --to ipynb them first, but that makes the experience worse than it was. Or more generally, I don't really understand the problem this PR is solving or what value it adds.

My main problem with the documentation is that we don't run the notebooks when publishing the docs, but at the same time we also don't have (all) the notebooks pre-rendered in the repository. Changing the notebooks to markdown form even makes it impossible to have their output included with them.

So to me, we should only convert the notebooks to markdown format when we actually have a mechanism in place to convert them to notebooks with output for the documentation. And to 'normal notebooks' for packaging and release. And at that point I don't really see the benefit of converting the notebooks to markdown; the notebooks without output would seem good enough for me.

What am I missing?

[teutoburg]

I can't just open them in jupyter notebook.

I was somewhat confused because precisely this worked nicely on my end. I realised this is because of some plugin to Jupyter. I'm not entirely certain because I have many things installed now, but I'm pretty sure it should be sufficient to have jupytext installed in the same environment that Jupyter is run from. Then the md-based notebooks should show up just like a "normal" ipynb one. I can also edit it there and it is saved back to the md format, without any need for manually running a conversion. This is of course without any output.

My main problem with the documentation is that we don't run the notebooks when publishing the docs

We would do just that in the new setup. That is, for the notebooks that are "cheap" enough to run them on RTD. For everything else that isn't, my plan was to keep them as .ipynb, run them regularly (on Zeus) and commit them with output. Then in my opinion we'll get the best of both worlds: Freshly run notebooks for every docs build without having to care about them for the simple ones, and pre-run static ones for the performance-heavy (but interesting) ones. Maybe those should at some point include information on which versions of everything were used to create them, so it is still somewhat reproducible.

In general I'm really not a fan of the .ipynb format for anything that get committed, because the diffs are a pain. For the "heavy" cases, I don't have a better option now, so those are out of the question anyway. But a lot of the lightweight ones are more like "pages in the documentation" in my opinion, rather than "example uses of ScopSim", so I think users are more likely to just copy out some of the relevant code, and not download and run the notebook as a whole (it's a different story for the "large science case" notebooks though, which we can keep as .ipynb). In the future, it will be possible to download an .ipynb file from the md-based notebooks anyway, once the sphinx book theme fixes that bug.