Fix pynest api generation for documentation using mock #1274

jessica-mitchell · 2019-08-27T08:51:24Z

This PR provides a solution for the PyNEST api generation on readthedocs.

The problem: pynest apis do not work on readthedocs.
For sphinx to auto generate the apis (sphinx-autodoc) for the website, it requires a running instance of nest. If we try to include nest in the build on ReadtheDocs, it times out because of resource limitations.

The solution:
Mock nest
All the credit goes to @Silmathoron, who provided the scripts/code required to mock nest!

The PyNEST apis functions were cleaned up a bit then indexed, and referenced in the docs

See output here: https://nest-test.readthedocs.io/en/pynest_mock/ref_material/pynest_apis.html

Mock pynestkernel

Add exceptions file

remove some content in connect function, provide link to connection managment guide, add func link and minor formatting changes removed old connecton info, update syntax and fix formatting issues remove whitespace, rtd req, add topology module add py and cpp to ignore and update headings

terhorstd · 2019-09-02T10:17:09Z

we decided to put this on hold for two weeks, because other PRs handle similar Python issues...

Silmathoron · 2019-09-02T11:42:07Z

@terhorstd sorry, this is a little unclear to me: which PRs are these? Which Python issues?

jessica-mitchell · 2019-09-02T12:00:35Z

@Silmathoron The issue he's referring to is the failing check on Travis with Python 2.7

Silmathoron · 2019-11-11T11:17:28Z

I just checked and it seems that the errors should be easy to fixed: mostly copyright header errors and maybe some other formatting errors that should be fixed automatically by using extras/check_copyright_headers.py and pep8fying

MSGBLD0220: Python files with formatting errors:
	MSGBLD0220: ... doc/conf.py
	MSGBLD0220: ... doc/pynestkernel_mock.py
	MSGBLD0220: ... pynest/nest/lib/hl_api_connections.py
	MSGBLD0220: ... pynest/nest/lib/hl_api_exceptions.py

pynest/nest/lib/hl_api_exceptions.py

Co-Authored-By: Tanguy Fardet <Silmathoron@users.noreply.github.com>

Silmathoron · 2019-11-13T16:31:26Z

Wow,
apparently the metaclass argument in Py3 is not only the metaclass but also inheritance I guess...
Is the metaclass really necessary? Can you check whether simple inheritance is sufficient?
@jessica-mitchell

jessica-mitchell · 2019-11-15T07:30:04Z

@Silmathoron I'm afraid my Python-fu is fairly limited - and looking at the code I am not really confident with what I would all need to change to test if simple inheritance would work.

Silmathoron · 2019-11-15T07:35:46Z

OK, I'll try to check it at some point in the week-end or next week.

Pynest mock Python 2 compatibility

jessica-mitchell · 2019-11-24T13:22:37Z

@Silmathoron During our 'docathon' thanks to @hakonsbm, we were able to solve the issues with Travis - could you take a look and approve this PR if there is nothing else we need?

Silmathoron

Nice work! Sorry I did not have time to look into it before in the end.
I'm wondering about the choice of always using the mocked kernel.
Otherwise, if possible I would generalize the use of references using the :func:`function_name nomenclature in the .rst files

Silmathoron · 2019-11-25T11:01:59Z

doc/conf.py

@@ -31,20 +31,55 @@
 import sys
 import os

+# import sphinx_gallery


remove this line

Silmathoron · 2019-11-25T11:04:02Z

doc/conf.py

+
+# -- Mock pynestkernel ----------------------------------------------------
+# The mock_kernel has to be imported after setting the correct sys paths.
+from mock_kernel import convert  # noqa


why not use the proper kernel if the install is not done on RtD?
If this really is a feature, then remove following comment lines

We were thinking for consistency, just leaving the mock_kernel for non RtD builds too; it then allows the docs to build without a proper nest install - Not sure if that's a 'feature', but could be useful.

OK, I don't have any preference, but if we keep the mock_kernel for all builds, then comments should go away ;)

Silmathoron · 2019-11-25T11:04:46Z

doc/conf.py

@@ -218,8 +265,6 @@ def setup(app):
 ]

 # -- Options for readthedocs ----------------------------------------------
-# on_rtd = os.environ.get('READTHEDOCS') == 'True'
-# if on_rtd:
 #    html_theme = 'alabaster'


also remove these lines

Silmathoron · 2019-11-25T11:06:50Z

doc/guides/connection_management.rst

 `SLI <an-introduction-to-sli.md>`__ ,the old syntax of the function
 still works, while in `PyNEST <introduction-to-pynest.md>`__, the
-``Connect()`` function has been renamed to ``OneToOneConnect()``.
+`Connect()` function has been renamed to `OneToOneConnect()`.


for existing functions, it would be nice to use the proper form to link directly to the corresponding docstring
either through :func:`nest.Connect or :func:`nest.lib.hl_api_connections.Connect depending on whether or not the full path is necessary.
do this to all functions in the API if possible

For creating these links, I'd like to wait for another PR -- we have a script that can search through all function occurrences and replace with the proper format.

Silmathoron · 2019-11-25T11:11:53Z

doc/ref_material/index.rst


 * The `Command Index <https://www.nest-simulator.org/helpindex/>`_ contains a list of all SLI and C++ related reference material.
+


remove extra blank lines

Silmathoron · 2019-11-25T11:25:06Z

pynest/nest/lib/hl_api_connections.py

@@ -522,7 +475,7 @@ def CGParse(xml_filename):
    generator cg.

    The library to provide the parsing can be selected
-    by CGSelectImplementation().
+    by :py:func:`.CGSelectImplementation`.


OK, if this link works, use this format for all links in previous .rst files (not sure the initial :py is actually necessary, is it?)

Silmathoron

OK, I just compiled the docs on my computer without issues, except for the fact that sphinx_gallery and sphinx_tabs are necessary and crash the build if not there... we should either mention it somewhere or make sure that a partial docs build is still done even if they are not there.

Apart from that, I would think we should make the PyNEST API more easy to find (Reference material > PyNEST functions) does not quite cut it for me (it is usually one of the first thing I am looking for in a documentaion).

Finally, I the API, I would put the "helper functions" at the very end and add a

.. contents::
   :local:

at the beginning

jessica-mitchell · 2019-11-26T10:23:00Z

@Silmathoron Thanks for your input! I updated the pynest apis so they appear on the side bar, they have an internal toc and are rearranged.

As for the sphinx_gallery and tabs - these should be listed in the requirements.txt file in doc/. (We still need to update this and the doc README) in a separate PR.

Silmathoron

Nice! thanks for the hard work 👍

steffengraber

Very nice work. I tested it and it works great. Just a few small comments.

steffengraber · 2019-11-26T14:03:07Z

doc/mock_kernel.py

+'''
+Mock pynestkernel.pyx into dummy python file.
+'''


Tripel double-quoted strings should be used for docstrings and a title is missing.

steffengraber · 2019-11-26T14:03:59Z

doc/mock_kernel.py

+
+
+def convert(infile):
+    '''


See my comment above.

jessica-mitchell and others added 3 commits August 27, 2019 10:02

Merge pull request #2 from Silmathoron/fix_pynest_apis2

3b6800d

Mock pynestkernel

Merge pull request #5 from Silmathoron/fix_pynest_apis2

cd6598e

Add exceptions file

terhorstd requested a review from Silmathoron October 14, 2019 10:06

Merge branch 'master' into pynest_mock

699d031

pep 8

1c0f679

Silmathoron reviewed Nov 13, 2019

View reviewed changes

pynest/nest/lib/hl_api_exceptions.py Outdated Show resolved Hide resolved

Update pynest/nest/lib/hl_api_exceptions.py

e3e3266

Co-Authored-By: Tanguy Fardet <Silmathoron@users.noreply.github.com>

jessica-mitchell mentioned this pull request Nov 23, 2019

Update syntax formatting for terms in documentation #1343

Closed

hakonsbm and others added 6 commits November 23, 2019 14:23

Fixed imports of mocking tools

237f40a

Removed generated mocked pynestkernel

58889e8

Made NESTErrors metaclass specification Python 2 and 3 compatible

f9edf1b

Merge pull request #9 from hakonsbm/pynest_mock

383014b

Pynest mock Python 2 compatibility

Merge branch 'master' into pynest_mock

9f4a930

Merge remote-tracking branch 'origin/pynest_mock' into pynest_mock

ebb159f

jessica-mitchell added ZP: PR Created DO NOT USE THIS LABEL and removed P: Blocked Work on this can not continue, see comments for the particular reason labels Nov 24, 2019

Merge branch 'master' into pynest_mock

219c203

Silmathoron suggested changes Nov 25, 2019

View reviewed changes

jessica-mitchell added 2 commits November 25, 2019 13:35

remove blank and commented out lines

afbe658

pep8

8dee2ab

Silmathoron suggested changes Nov 25, 2019

View reviewed changes

jessica-mitchell added 2 commits November 25, 2019 15:24

change toc from index to pynest_apis

5d684e1

rearrange pynest apis, add to toc

c3a2ee5

Silmathoron approved these changes Nov 26, 2019

View reviewed changes

heplesser requested a review from steffengraber November 26, 2019 10:39

steffengraber suggested changes Nov 26, 2019

View reviewed changes

fix quotes

e3e5b64

steffengraber approved these changes Nov 26, 2019

View reviewed changes

hakonsbm merged commit 5a79531 into nest:master Nov 27, 2019

jessica-mitchell deleted the pynest_mock branch March 30, 2021 08:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix pynest api generation for documentation using mock #1274

Fix pynest api generation for documentation using mock #1274

jessica-mitchell commented Aug 27, 2019

terhorstd commented Sep 2, 2019

Silmathoron commented Sep 2, 2019

jessica-mitchell commented Sep 2, 2019

Silmathoron commented Nov 11, 2019

Silmathoron commented Nov 13, 2019

jessica-mitchell commented Nov 15, 2019

Silmathoron commented Nov 15, 2019

jessica-mitchell commented Nov 24, 2019

Silmathoron left a comment

Silmathoron Nov 25, 2019

Silmathoron Nov 25, 2019

jessica-mitchell Nov 25, 2019

Silmathoron Nov 25, 2019

jessica-mitchell Nov 25, 2019

Silmathoron Nov 25, 2019

Silmathoron Nov 25, 2019

jessica-mitchell Nov 25, 2019

Silmathoron Nov 25, 2019

Silmathoron Nov 25, 2019

Silmathoron left a comment

jessica-mitchell commented Nov 26, 2019

Silmathoron left a comment

steffengraber left a comment

steffengraber Nov 26, 2019

steffengraber Nov 26, 2019


		* The `Command Index <https://www.nest-simulator.org/helpindex/>`_ contains a list of all SLI and C++ related reference material.

Fix pynest api generation for documentation using mock #1274

Fix pynest api generation for documentation using mock #1274

Conversation

jessica-mitchell commented Aug 27, 2019

terhorstd commented Sep 2, 2019

Silmathoron commented Sep 2, 2019

jessica-mitchell commented Sep 2, 2019

Silmathoron commented Nov 11, 2019

Silmathoron commented Nov 13, 2019

jessica-mitchell commented Nov 15, 2019

Silmathoron commented Nov 15, 2019

jessica-mitchell commented Nov 24, 2019

Silmathoron left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Silmathoron left a comment

Choose a reason for hiding this comment

jessica-mitchell commented Nov 26, 2019

Silmathoron left a comment

Choose a reason for hiding this comment

steffengraber left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment