# gammapy/gammapy

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

# Finish integration of Jupyter notebooks with Sphinx docs #1215

Merged
merged 18 commits into from Nov 22, 2017

## Conversation

Projects
None yet
2 participants
Member

### Bultako commented Nov 20, 2017

 $setup.py build_docs added flag execute_notebooks = False in setup.cfg sets no execution of notebooks during the docs building process added flag rebuild_notebooks = True in setup.cfg sets always replace notebooks from$GAMMAPY_EXTRA and rebuild copies notebooks from \$GAMMAPY_EXTRA to _static/notebooks folder, so RST files, ipynb notebooks, and sphinx formatted notebooks live in the same version/tag links added in fixed-text sphinx formatted notebooks for downloading live *.ipynb notebooks transforms absolute links from notebooks to http://docs.gammapy.org website into relative links to RST files, so RST files, ipynb notebooks, and sphinx formatted notebooks live in the same version/tag added some documentation related with these modifs previous work: #1176 see generated doc: http://www.iaa.es/~jer/gammapydocs/notebooks/first_steps.html

### Bultako added some commits Nov 17, 2017

 Copy notebooks from to _static/notebooks folder 
 fb6aeea 
 Prune docs/_static/notebooks 
 87a747d 
 Add option in configuration for rebuilding sphinx notebooks 
 70523ab 
 Add option in configuration for executing sphinx notebooks 
 ec66ec9 
 sync versions between notebooks and rst files 
regexp in cells replacing absolute links to http://docs.gammapy.org/en/latest into relative
links present in raw notebooks (.ipynb files) and sphinx notebooks (.html formatted notebooks)
relative links guarantee pointing to the same old doc version, not necessary the latest
 5bf7972 
 Fix warning ‘reference not found’ 
Links from notebooks pointing to ../datasets/index.rst#gammapy-extra are not found raising a warning.
 e3bf817 
 Added links download notebook 
 6b555fa 
 Add documentation related with Sphinx notebooks 
 1e6c0c1 

Member

### cdeil commented Nov 21, 2017

@Bultako - Thanks!

The main thing before this can go in is that the docs build in our CI has to be green.
I'm not 100% sure, but I think it's failing because of these new warnings:
https://travis-ci.org/gammapy/gammapy/jobs/304759871#L1953
(we run with build_docs -w to turn warnings into errors to make the CI build fail, see here).

One suggestion I have would be to move the code you've added to docs/conf.py mostly out to gammapy/utils, like we did here:

Line 189 in ea9dcde

 from gammapy.utils.docs import gammapy_sphinx_ext_activate

The problem with putting more and more code in docs/conf.py is that it's hard to maintain (i.e. read / test) because docs/conf.py is already long and spaghetti code with a lot of Sphinx-specific stuff. So I think we should try to avoid it growing much and instead put as much code as possible in modules / functions / classes (whatever is most convenient) in gammapy/utils so that it becomes more readable (and in the future maybe even testable, although for now I don't think we should try to add tests -- the real docs build is the test we have and need).

I'll have a closer look now and leave some inline comments.

### cdeil reviewed Nov 21, 2017

docs/conf.py Outdated

### cdeil reviewed Nov 21, 2017

docs/conf.py Outdated

### cdeil reviewed Nov 21, 2017

docs/conf.py Outdated

### cdeil reviewed Nov 21, 2017

docs/conf.py Outdated
Member

### cdeil commented Nov 21, 2017

 @Bultako - I've left a few inline comments. I like how you've introduced the info box at the top with a download link for the notebook! I have a few suggestions that I think would make it more intuitive for users: For the "download notebook" link, I think it would be great to change to "Download notebook: nbfile.ipynb" and to just have the "nbfile.ipynb" be the link. This would IMO make it more intuitive for users which file they will download and then will have to look for locally to execute. I also think it would be good to add the filenames to every notebook linked to from this page: http://docs.gammapy.org/en/latest/notebooks.html#notebooks again because people will have to know the filenames of the notebooks to start executing them and recognise them. For the description you have for downloading all HTML locally, I think it would be better if you put a link "Download this version of the docs with all notebooks: gammapy.zip" and have the filename be a link that downloads http://readthedocs.org/projects/gammapy/downloads/htmlzip/latest/ . So don't rely on people finding the javascript applet from RTD and the "HTML" link inside that. Since this part will be the same for all notebooks, and then users will need some more info how to find and start the notebooks, that could also go on a separate docs page and then just link to it from the info box. As you like for now. Finally, on nbviewer (e.g. https://nbviewer.jupyter.org/github/gammapy/gammapy-extra/blob/master/notebooks/cta_1dc_introduction.ipynb) in the top right corner there's two more useful links: one to the notebook on Github (i.e. the file someone would have to edit if they wanted to contribute) and one called "view as code" that is just a text file that people could copy & paste from into a Python script. Both of those aren't essential, but if they are easy to do, I think it would be useful to have. @Bultako - The big extra feature would be if people could execute the examples on Binder. Axel and I tried quite a bit in the past, but mostly failed to get it to run properly. If you are interested to add it, please have a look at #726 and comment there if you want to try setting it up again for Gammapy. (Let's keep that separate from this PR, to avoid it getting very large or stalled).

### Bultako added some commits Nov 21, 2017

 Fix warnings in travis CI 
 26a4a2e 
 Generate .py scripts from notebooks 
 a485d97 
Member

### cdeil commented Nov 21, 2017

 @Bultako - Locally I always see these three warnings on python setup.py build_docs: https://gist.github.com/cdeil/9419c8d57ecf7f2e4bc90522757d942e#file-gistfile1-txt-L485 /Users/deil/temp/gammapy/docs/notebooks/cta_1dc_introduction.ipynb:9: WARNING: Explicit markup ends without a blank line; unexpected unindent. /Users/deil/temp/gammapy/docs/notebooks/cta_1dc_introduction.ipynb:1657: WARNING: Explicit markup ends without a blank line; unexpected unindent. /Users/deil/temp/gammapy/docs/notebooks/cta_data_analysis.ipynb:9: WARNING: Explicit markup ends without a blank line; unexpected unindent.  This is on a clean gammapy git checkout - it shouldn't have anything to do with me having old build files lying around. I don't understand why these warnings appear for me, but not on travis-ci (see https://travis-ci.org/gammapy/gammapy/jobs/305233458). !?
Member

### Bultako commented Nov 21, 2017 • edited

 @cdeil we are using notebooks from last version of gammapy-extragithub repo so I guess you should check if you are in a clean gammapy-extra git checkout -
Member

### cdeil commented Nov 21, 2017

 My gammapy-extra is at fe9daa9b59d97570175b85803bc0bf2d11037260 which is up-to-date as of today: https://github.com/gammapy/gammapy-extra/commits/master Also git status there doesn't show any modifications and I also did git clean -fdx in the gammapy-extra repo just to be sure. I really don't understand this. The only thing I can think of are version differences between my machine (e.g. in nbconvert)!?
Member

### Bultako commented Nov 21, 2017

 @cdeil I would check the version of nbsphinx .travis.yml is using last version of nbsphinx 0.2.17 installed with pip https://github.com/Bultako/gammapy/blob/26a4a2e9ca08d316705a8199478a90688a5565b2/.travis.yml#L38 conda-forge provides an older version that does not handle links with RST files properly
Member

### cdeil commented Nov 21, 2017

 I updated nbsphinx, but I still see these three warnings. This is what I have now: In [2]: nbsphinx.__version__ Out[2]: '0.2.17' In [4]: nbconvert.__version__ Out[4]: '5.3.1' In [6]: nbformat.__version__ Out[6]: '4.4.0' 
Member

### Bultako commented Nov 21, 2017

 just for info: In [5]: nbsphinx.__version__ Out[5]: '0.2.17' In [6]: nbconvert.__version__ Out[6]: '5.2.1' In [7]: nbformat.__version__ Out[7]: '4.3.0'  missing a shared docker container for reproducibility. when conda config YAML files for virtual envs are not enough. I'll have a deeper look later.

### Bultako added some commits Nov 21, 2017

 Add filenames to the list of notebooks 
 08818a1 
 Refactoring code into utils/docs.py 
 b45ebdd 
 Fix execute_notebooks config option 
 4238e07 
 Fix clean_notebooks config option 
 8862a20 
 Fix box template var replacement 
 1704c20 
 Modify box text/links and minor review changes 
 71c6830 
Member

### Bultako commented Nov 21, 2017 • edited

 @cdeil I think I've fixed all your review comments, you can see the new generated docs here: http://www.iaa.es/~jer/gammapydocs/notebooks.html I suspect some more fixes may be needed so RTD could serve *.ipynb and *.py files as downloadable files. in relation to your local warnings, not a clue :(
 Allows display/dowload of python scripts as text files 
 010f258 

### cdeil requested changes Nov 22, 2017

@Bultako - This looks very good!

I've put some minor code suggestions inline.

The download links look great now:

Source files: astropy_introduction.ipynb | astropy_introduction.py


Currently clicking the links doesn't download a file, but opens up the file as text in the browser. Would it be possible and preferable to have the default behaviour on click be a download? I'm not sure it's possible, but with this or by putting a raw HTML a tag with download attribute like this it might.
Alternatively: maybe add something like this on the line below:

(to download the files to your local machine: right-click and select "save as")

 @@ -88,7 +88,8 @@ extensions.append('nbsphinx') extensions.append('IPython.sphinxext.ipython_console_highlighting') extensions.append('sphinx.ext.mathjax') nbsphinx_execute = 'never' nbsphinx_execute = setup_cfg['execute_notebooks']

#### cdeil Nov 22, 2017

Member

This line can now be removed from conf.py, because that local variable is now unused, no?

#### Bultako Nov 22, 2017

Member

we actually need it in conf.py if we want to manage the option to enable/disable the execution of notebooks during the process of doc generation
http://nbsphinx.readthedocs.io/en/0.2.9/never-execute.html

we can also remove it from step.cfg and leave it as hard text in conf.py

#### cdeil Nov 22, 2017

Member

Ah, OK. Could you please add a one-line comment?

# Configure whether to run nbsphinx; see http://nbsphinx.readthedocs.io/en/stable/never-execute.html


I think leaving this configurable via setup.cfg is better, no?

#### Bultako Nov 22, 2017

Member

yes, I would leave it in setup.cfg since other config options (also the ones used for doc generation) are living there.

docs/development/howto.rst Outdated
gammapy/utils/docs.py Outdated
gammapy/utils/docs.py Outdated
gammapy/utils/docs.py Outdated
 Fix minor review comments 
 708d853 
Member

### Bultako commented Nov 22, 2017

 @cdeil re: forcing download in links I don't know how to do this. The point is that nb_sphinx makes the conversion from .ipynb to .rst files before actually executing Sphinx, transforming markdown syntax for links in notebooks to rst syntax for links. I have no access to these intermediate short-living rst files, ipynb are converted to html directly. This does not allow any simple way for fine-tuning links and add a download attribute to a hrefs, neither the :download: role added in the .ipynb file seems to work. For the moment I've just added the message right-click and select "save as"
Member

### cdeil commented Nov 22, 2017

 OK. Ready to merge?
Member

 Ready

### cdeil merged commit af4c7a9 into gammapy:master Nov 22, 2017 1 of 2 checks passed

#### 1 of 2 checks passed

continuous-integration/travis-ci/pr The Travis CI build is in progress
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
Member

### cdeil commented Nov 22, 2017

 Thank you!

### cdeil referenced this pull request Apr 5, 2018

Closed

#### Sphinx warning from CTA notebooks #1361

to join this conversation on GitHub. Already have an account? Sign in to comment