New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Finish integration of Jupyter notebooks with Sphinx docs #1215

Merged
merged 18 commits into from Nov 22, 2017

Conversation

Projects
None yet
2 participants
@Bultako
Member

Bultako commented Nov 20, 2017

$ setup.py build_docs

  • added flag execute_notebooks = False in setup.cfg sets no execution of notebooks during the docs building process
  • added flag rebuild_notebooks = True in setup.cfg sets always replace notebooks from $GAMMAPY_EXTRA and rebuild
  • copies notebooks from $GAMMAPY_EXTRA to _static/notebooks folder, so RST files, ipynb notebooks, and sphinx formatted notebooks live in the same version/tag
  • links added in fixed-text sphinx formatted notebooks for downloading live *.ipynb notebooks
  • transforms absolute links from notebooks to http://docs.gammapy.org website into relative links to RST files, so RST files, ipynb notebooks, and sphinx formatted notebooks live in the same version/tag
  • added some documentation related with these modifs

previous work:
#1176

see generated doc:
http://www.iaa.es/~jer/gammapydocs/notebooks/first_steps.html

Bultako added some commits Nov 17, 2017

sync versions between notebooks and rst files
regexp in cells replacing absolute links to http://docs.gammapy.org/en/latest into relative
links present in raw notebooks (.ipynb files) and sphinx notebooks (.html formatted notebooks)
relative links guarantee pointing to the same old doc version, not necessary the latest
Fix warning ‘reference not found’
Links from notebooks pointing to ../datasets/index.rst#gammapy-extra are not found raising a warning.

@cdeil cdeil self-assigned this Nov 21, 2017

@cdeil cdeil added the docs label Nov 21, 2017

@cdeil cdeil added this to the 0.7 milestone Nov 21, 2017

@cdeil

This comment has been minimized.

Show comment
Hide comment
@cdeil

cdeil Nov 21, 2017

Member

@Bultako - Thanks!

The main thing before this can go in is that the docs build in our CI has to be green.
I'm not 100% sure, but I think it's failing because of these new warnings:
https://travis-ci.org/gammapy/gammapy/jobs/304759871#L1953
(we run with build_docs -w to turn warnings into errors to make the CI build fail, see here).

One suggestion I have would be to move the code you've added to docs/conf.py mostly out to gammapy/utils, like we did here:

from gammapy.utils.docs import gammapy_sphinx_ext_activate

The problem with putting more and more code in docs/conf.py is that it's hard to maintain (i.e. read / test) because docs/conf.py is already long and spaghetti code with a lot of Sphinx-specific stuff. So I think we should try to avoid it growing much and instead put as much code as possible in modules / functions / classes (whatever is most convenient) in gammapy/utils so that it becomes more readable (and in the future maybe even testable, although for now I don't think we should try to add tests -- the real docs build is the test we have and need).

I'll have a closer look now and leave some inline comments.

Member

cdeil commented Nov 21, 2017

@Bultako - Thanks!

The main thing before this can go in is that the docs build in our CI has to be green.
I'm not 100% sure, but I think it's failing because of these new warnings:
https://travis-ci.org/gammapy/gammapy/jobs/304759871#L1953
(we run with build_docs -w to turn warnings into errors to make the CI build fail, see here).

One suggestion I have would be to move the code you've added to docs/conf.py mostly out to gammapy/utils, like we did here:

from gammapy.utils.docs import gammapy_sphinx_ext_activate

The problem with putting more and more code in docs/conf.py is that it's hard to maintain (i.e. read / test) because docs/conf.py is already long and spaghetti code with a lot of Sphinx-specific stuff. So I think we should try to avoid it growing much and instead put as much code as possible in modules / functions / classes (whatever is most convenient) in gammapy/utils so that it becomes more readable (and in the future maybe even testable, although for now I don't think we should try to add tests -- the real docs build is the test we have and need).

I'll have a closer look now and leave some inline comments.

Show outdated Hide outdated docs/conf.py Outdated
Show outdated Hide outdated docs/conf.py Outdated
Show outdated Hide outdated docs/conf.py Outdated
Show outdated Hide outdated docs/conf.py Outdated
@cdeil

This comment has been minimized.

Show comment
Hide comment
@cdeil

cdeil Nov 21, 2017

Member

@Bultako - I've left a few inline comments.

I like how you've introduced the info box at the top with a download link for the notebook!

screen shot 2017-11-21 at 10 55 02

I have a few suggestions that I think would make it more intuitive for users:

For the "download notebook" link, I think it would be great to change to "Download notebook: nbfile.ipynb" and to just have the "nbfile.ipynb" be the link. This would IMO make it more intuitive for users which file they will download and then will have to look for locally to execute. I also think it would be good to add the filenames to every notebook linked to from this page: http://docs.gammapy.org/en/latest/notebooks.html#notebooks again because people will have to know the filenames of the notebooks to start executing them and recognise them.

For the description you have for downloading all HTML locally, I think it would be better if you put a link "Download this version of the docs with all notebooks: gammapy.zip" and have the filename be a link that downloads http://readthedocs.org/projects/gammapy/downloads/htmlzip/latest/ . So don't rely on people finding the javascript applet from RTD and the "HTML" link inside that. Since this part will be the same for all notebooks, and then users will need some more info how to find and start the notebooks, that could also go on a separate docs page and then just link to it from the info box. As you like for now.

Finally, on nbviewer (e.g. https://nbviewer.jupyter.org/github/gammapy/gammapy-extra/blob/master/notebooks/cta_1dc_introduction.ipynb) in the top right corner there's two more useful links: one to the notebook on Github (i.e. the file someone would have to edit if they wanted to contribute) and one called "view as code" that is just a text file that people could copy & paste from into a Python script. Both of those aren't essential, but if they are easy to do, I think it would be useful to have.

@Bultako - The big extra feature would be if people could execute the examples on Binder. Axel and I tried quite a bit in the past, but mostly failed to get it to run properly. If you are interested to add it, please have a look at #726 and comment there if you want to try setting it up again for Gammapy. (Let's keep that separate from this PR, to avoid it getting very large or stalled).

Member

cdeil commented Nov 21, 2017

@Bultako - I've left a few inline comments.

I like how you've introduced the info box at the top with a download link for the notebook!

screen shot 2017-11-21 at 10 55 02

I have a few suggestions that I think would make it more intuitive for users:

For the "download notebook" link, I think it would be great to change to "Download notebook: nbfile.ipynb" and to just have the "nbfile.ipynb" be the link. This would IMO make it more intuitive for users which file they will download and then will have to look for locally to execute. I also think it would be good to add the filenames to every notebook linked to from this page: http://docs.gammapy.org/en/latest/notebooks.html#notebooks again because people will have to know the filenames of the notebooks to start executing them and recognise them.

For the description you have for downloading all HTML locally, I think it would be better if you put a link "Download this version of the docs with all notebooks: gammapy.zip" and have the filename be a link that downloads http://readthedocs.org/projects/gammapy/downloads/htmlzip/latest/ . So don't rely on people finding the javascript applet from RTD and the "HTML" link inside that. Since this part will be the same for all notebooks, and then users will need some more info how to find and start the notebooks, that could also go on a separate docs page and then just link to it from the info box. As you like for now.

Finally, on nbviewer (e.g. https://nbviewer.jupyter.org/github/gammapy/gammapy-extra/blob/master/notebooks/cta_1dc_introduction.ipynb) in the top right corner there's two more useful links: one to the notebook on Github (i.e. the file someone would have to edit if they wanted to contribute) and one called "view as code" that is just a text file that people could copy & paste from into a Python script. Both of those aren't essential, but if they are easy to do, I think it would be useful to have.

@Bultako - The big extra feature would be if people could execute the examples on Binder. Axel and I tried quite a bit in the past, but mostly failed to get it to run properly. If you are interested to add it, please have a look at #726 and comment there if you want to try setting it up again for Gammapy. (Let's keep that separate from this PR, to avoid it getting very large or stalled).

@cdeil

This comment has been minimized.

Show comment
Hide comment
@cdeil

cdeil Nov 21, 2017

Member

@Bultako - Locally I always see these three warnings on python setup.py build_docs:
https://gist.github.com/cdeil/9419c8d57ecf7f2e4bc90522757d942e#file-gistfile1-txt-L485

/Users/deil/temp/gammapy/docs/notebooks/cta_1dc_introduction.ipynb:9: WARNING: Explicit markup ends without a blank line; unexpected unindent.
/Users/deil/temp/gammapy/docs/notebooks/cta_1dc_introduction.ipynb:1657: WARNING: Explicit markup ends without a blank line; unexpected unindent.
/Users/deil/temp/gammapy/docs/notebooks/cta_data_analysis.ipynb:9: WARNING: Explicit markup ends without a blank line; unexpected unindent.

This is on a clean gammapy git checkout - it shouldn't have anything to do with me having old build files lying around. I don't understand why these warnings appear for me, but not on travis-ci (see https://travis-ci.org/gammapy/gammapy/jobs/305233458).

!?

Member

cdeil commented Nov 21, 2017

@Bultako - Locally I always see these three warnings on python setup.py build_docs:
https://gist.github.com/cdeil/9419c8d57ecf7f2e4bc90522757d942e#file-gistfile1-txt-L485

/Users/deil/temp/gammapy/docs/notebooks/cta_1dc_introduction.ipynb:9: WARNING: Explicit markup ends without a blank line; unexpected unindent.
/Users/deil/temp/gammapy/docs/notebooks/cta_1dc_introduction.ipynb:1657: WARNING: Explicit markup ends without a blank line; unexpected unindent.
/Users/deil/temp/gammapy/docs/notebooks/cta_data_analysis.ipynb:9: WARNING: Explicit markup ends without a blank line; unexpected unindent.

This is on a clean gammapy git checkout - it shouldn't have anything to do with me having old build files lying around. I don't understand why these warnings appear for me, but not on travis-ci (see https://travis-ci.org/gammapy/gammapy/jobs/305233458).

!?

@Bultako

This comment has been minimized.

Show comment
Hide comment
@Bultako

Bultako Nov 21, 2017

Member

@cdeil
we are using notebooks from last version of gammapy-extragithub repo
so I guess you should check if you are in a clean gammapy-extra git checkout -

Member

Bultako commented Nov 21, 2017

@cdeil
we are using notebooks from last version of gammapy-extragithub repo
so I guess you should check if you are in a clean gammapy-extra git checkout -

@cdeil

This comment has been minimized.

Show comment
Hide comment
@cdeil

cdeil Nov 21, 2017

Member

My gammapy-extra is at fe9daa9b59d97570175b85803bc0bf2d11037260 which is up-to-date as of today: https://github.com/gammapy/gammapy-extra/commits/master

Also git status there doesn't show any modifications and I also did git clean -fdx in the gammapy-extra repo just to be sure.

I really don't understand this. The only thing I can think of are version differences between my machine (e.g. in nbconvert)!?

Member

cdeil commented Nov 21, 2017

My gammapy-extra is at fe9daa9b59d97570175b85803bc0bf2d11037260 which is up-to-date as of today: https://github.com/gammapy/gammapy-extra/commits/master

Also git status there doesn't show any modifications and I also did git clean -fdx in the gammapy-extra repo just to be sure.

I really don't understand this. The only thing I can think of are version differences between my machine (e.g. in nbconvert)!?

@Bultako

This comment has been minimized.

Show comment
Hide comment
@Bultako

Bultako Nov 21, 2017

Member

@cdeil
I would check the version of nbsphinx
.travis.yml is using last version of nbsphinx 0.2.17 installed with pip

https://github.com/Bultako/gammapy/blob/26a4a2e9ca08d316705a8199478a90688a5565b2/.travis.yml#L38

conda-forge provides an older version that does not handle links with RST files properly

Member

Bultako commented Nov 21, 2017

@cdeil
I would check the version of nbsphinx
.travis.yml is using last version of nbsphinx 0.2.17 installed with pip

https://github.com/Bultako/gammapy/blob/26a4a2e9ca08d316705a8199478a90688a5565b2/.travis.yml#L38

conda-forge provides an older version that does not handle links with RST files properly

@cdeil

This comment has been minimized.

Show comment
Hide comment
@cdeil

cdeil Nov 21, 2017

Member

I updated nbsphinx, but I still see these three warnings. This is what I have now:

In [2]: nbsphinx.__version__
Out[2]: '0.2.17'

In [4]: nbconvert.__version__
Out[4]: '5.3.1'

In [6]: nbformat.__version__
Out[6]: '4.4.0'
Member

cdeil commented Nov 21, 2017

I updated nbsphinx, but I still see these three warnings. This is what I have now:

In [2]: nbsphinx.__version__
Out[2]: '0.2.17'

In [4]: nbconvert.__version__
Out[4]: '5.3.1'

In [6]: nbformat.__version__
Out[6]: '4.4.0'
@Bultako

This comment has been minimized.

Show comment
Hide comment
@Bultako

Bultako Nov 21, 2017

Member

just for info:

In [5]: nbsphinx.__version__
Out[5]: '0.2.17'

In [6]: nbconvert.__version__
Out[6]: '5.2.1'

In [7]: nbformat.__version__    
Out[7]: '4.3.0'

missing a shared docker container for reproducibility.
when conda config YAML files for virtual envs are not enough.
I'll have a deeper look later.

Member

Bultako commented Nov 21, 2017

just for info:

In [5]: nbsphinx.__version__
Out[5]: '0.2.17'

In [6]: nbconvert.__version__
Out[6]: '5.2.1'

In [7]: nbformat.__version__    
Out[7]: '4.3.0'

missing a shared docker container for reproducibility.
when conda config YAML files for virtual envs are not enough.
I'll have a deeper look later.

@Bultako

This comment has been minimized.

Show comment
Hide comment
@Bultako

Bultako Nov 21, 2017

Member

@cdeil
I think I've fixed all your review comments, you can see the new generated docs here:
http://www.iaa.es/~jer/gammapydocs/notebooks.html

I suspect some more fixes may be needed so RTD could serve *.ipynb and *.py files as downloadable files.

in relation to your local warnings, not a clue :(

Member

Bultako commented Nov 21, 2017

@cdeil
I think I've fixed all your review comments, you can see the new generated docs here:
http://www.iaa.es/~jer/gammapydocs/notebooks.html

I suspect some more fixes may be needed so RTD could serve *.ipynb and *.py files as downloadable files.

in relation to your local warnings, not a clue :(

@cdeil

@Bultako - This looks very good!

I've put some minor code suggestions inline.

The download links look great now:

Source files: astropy_introduction.ipynb | astropy_introduction.py

Currently clicking the links doesn't download a file, but opens up the file as text in the browser. Would it be possible and preferable to have the default behaviour on click be a download? I'm not sure it's possible, but with this or by putting a raw HTML a tag with download attribute like this it might.
Alternatively: maybe add something like this on the line below:

(to download the files to your local machine: right-click and select "save as")
@@ -88,7 +88,8 @@
extensions.append('nbsphinx')
extensions.append('IPython.sphinxext.ipython_console_highlighting')
extensions.append('sphinx.ext.mathjax')
nbsphinx_execute = 'never'
nbsphinx_execute = setup_cfg['execute_notebooks']

This comment has been minimized.

@cdeil

cdeil Nov 22, 2017

Member

This line can now be removed from conf.py, because that local variable is now unused, no?

@cdeil

cdeil Nov 22, 2017

Member

This line can now be removed from conf.py, because that local variable is now unused, no?

This comment has been minimized.

@Bultako

Bultako Nov 22, 2017

Member

we actually need it in conf.py if we want to manage the option to enable/disable the execution of notebooks during the process of doc generation
http://nbsphinx.readthedocs.io/en/0.2.9/never-execute.html

we can also remove it from step.cfg and leave it as hard text in conf.py

@Bultako

Bultako Nov 22, 2017

Member

we actually need it in conf.py if we want to manage the option to enable/disable the execution of notebooks during the process of doc generation
http://nbsphinx.readthedocs.io/en/0.2.9/never-execute.html

we can also remove it from step.cfg and leave it as hard text in conf.py

This comment has been minimized.

@cdeil

cdeil Nov 22, 2017

Member

Ah, OK. Could you please add a one-line comment?

# Configure whether to run nbsphinx; see http://nbsphinx.readthedocs.io/en/stable/never-execute.html

I think leaving this configurable via setup.cfg is better, no?

@cdeil

cdeil Nov 22, 2017

Member

Ah, OK. Could you please add a one-line comment?

# Configure whether to run nbsphinx; see http://nbsphinx.readthedocs.io/en/stable/never-execute.html

I think leaving this configurable via setup.cfg is better, no?

This comment has been minimized.

@Bultako

Bultako Nov 22, 2017

Member

yes, I would leave it in setup.cfg since other config options (also the ones used for doc generation) are living there.

@Bultako

Bultako Nov 22, 2017

Member

yes, I would leave it in setup.cfg since other config options (also the ones used for doc generation) are living there.

Show outdated Hide outdated docs/development/howto.rst Outdated
Show outdated Hide outdated gammapy/utils/docs.py Outdated
Show outdated Hide outdated gammapy/utils/docs.py Outdated
Show outdated Hide outdated gammapy/utils/docs.py Outdated
@Bultako

This comment has been minimized.

Show comment
Hide comment
@Bultako

Bultako Nov 22, 2017

Member

@cdeil
re: forcing download in links

I don't know how to do this. The point is that nb_sphinx makes the conversion from .ipynb to .rst files before actually executing Sphinx, transforming markdown syntax for links in notebooks to rst syntax for links. I have no access to these intermediate short-living rst files, ipynb are converted to html directly. This does not allow any simple way for fine-tuning links and add a download attribute to a hrefs, neither the :download: role added in the .ipynb file seems to work.

For the moment I've just added the message right-click and select "save as"

Member

Bultako commented Nov 22, 2017

@cdeil
re: forcing download in links

I don't know how to do this. The point is that nb_sphinx makes the conversion from .ipynb to .rst files before actually executing Sphinx, transforming markdown syntax for links in notebooks to rst syntax for links. I have no access to these intermediate short-living rst files, ipynb are converted to html directly. This does not allow any simple way for fine-tuning links and add a download attribute to a hrefs, neither the :download: role added in the .ipynb file seems to work.

For the moment I've just added the message right-click and select "save as"

@cdeil

This comment has been minimized.

Show comment
Hide comment
@cdeil

cdeil Nov 22, 2017

Member

OK. Ready to merge?

Member

cdeil commented Nov 22, 2017

OK. Ready to merge?

@Bultako

This comment has been minimized.

Show comment
Hide comment
@Bultako

Bultako Nov 22, 2017

Member

Ready

Member

Bultako commented Nov 22, 2017

Ready

@cdeil

cdeil approved these changes Nov 22, 2017

@cdeil cdeil merged commit af4c7a9 into gammapy:master Nov 22, 2017

1 of 2 checks passed

continuous-integration/travis-ci/pr The Travis CI build is in progress
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
@cdeil

This comment has been minimized.

Show comment
Hide comment
@cdeil

cdeil Nov 22, 2017

Member

Thank you!

Member

cdeil commented Nov 22, 2017

Thank you!

@Bultako Bultako deleted the Bultako:nbsphinx branch Nov 22, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment