Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python 3: Restore developer documentation #9840

Closed
LeonarddeR opened this issue Jun 28, 2019 · 4 comments · Fixed by #9968
Closed

Python 3: Restore developer documentation #9840

LeonarddeR opened this issue Jun 28, 2019 · 4 comments · Fixed by #9968
Milestone

Comments

@LeonarddeR
Copy link
Collaborator

Having the ability to run scons on Python 3 made us demolish the ability to create Python source code developer documentation temporarily. This is because epydoc isn't supported on Python 3.

@LeonarddeR
Copy link
Collaborator Author

I've looked into using Sphinx. Some things I found:

  • Python, wxPython, PySerial and ConfigObj also use Sphinx
  • It is an advised tool by readthedocs
  • Installing it from pip requires some additional dependencies. It couldn't be expected from us that we bundle these all in git submodules. We could consider an additional binary repository, tough. I think this is preferred when we decide to use sphinx for more than just the developer documentation.
  • It uses RestructuredText as the main code style. It has support for markdown, though only using an extra package. RestructuredText has other benefits over markdown, such as keywords and tables in the default spec.
  • It supports type hints with an additional extension
  • There is extensive documentation, many products use it.
  • For our developer guide, it would be possible to link from the dev guide to module/class/function documentation.

In the current situation, we're using epytext/epydoc style doc strings. These are not compatible with sphinx by default. There is an additional module for this, which I haven't yet tested. I'm not convinced that this is the right direction though. May be we should convert everything to rst/sphinx native style. This shouldn't be very difficult. See this article about a possible migration strategy. There's also the Pyment which I've tested quickly, but that tool doesn't replace the L{link} style markup from epydoc and tends to replace tabs with spaces.

@LeonarddeR
Copy link
Collaborator Author

See also #3255, cross posting there.

In my sphinx branch I created a prototype:

  1. Install sphinx using pip: pip3 install sphinx
  2. In the mentioned branch, navigate to the devDocs directory
  3. Execute make html
  4. Consult the _build folder for the output.

Note that the doc string annotations are still epydoc style. This is also a reason for many warnings. However, I've been able to get rid of all tracebacks at source code parsing time.

Cc @feerrenrut

@LeonarddeR
Copy link
Collaborator Author

Investigation update.

After some more tests, it really looks like pyment has too much bugs to continue with it. It introduces empty lines, touches part of function bodies it really should leave alone, etc.
On the other hand, the approach with sed misses the fact that sphinx requires a blank line between the doc string body and the parameter definitions. I wonder whether we could automate this somehow.

@LeonarddeR
Copy link
Collaborator Author

I created a gist that contains prototype code to convert doc strings to sphinx style using regex.

https://gist.github.com/leonardder/82564a57a1a79a36fad79c1b92771c29

feerrenrut pushed a commit that referenced this issue Jan 20, 2020
Fixes #9840

Summary:

When we switched to Python 3, we had to abandon Epydoc for developer documentation.
From the alternatives, sphinx seemed to be the most useful to us:
- It is the only supported api doc generator on Read the Docs
- It is used by numerous other projects, including python, wxPython, pyserial and configobj
- It is heavily extensible.
- In the end, it might allow us to integrate the user docs in the build process, thereby adding the ability to abandon txt2tags. Even if we don't, sphinx has markdown support with a simple extension.

Implementation:

Reimplements the functionality of `scons devDocs`, they are now build using sphinx.
Move `devDocs` build script from `sconstruct` to the devDocs folder


Issues:
- The layout of the dev docs is likely to be broken, as we are still using epidoc style doc strings that have to be converted to the Python (restructured text) style doc strings
- Read the Docs builds docs on Linux. If we want to support this, it will require some source code changes. We can work around a lot of cases using mock imports. We may be able to build the docs ourselves on CI and push them afterwards. 
- No built in support for type hints, though there is an extension for that.

Incompatible doc string syntax:
Currently, we use epytext/epydoc style annotations for our doc strings. such as:
	@param test: a parameter
	@type test: int
Sphinx uses directives like `:param` and `:type`. It also allows nesting them, e.g. `:param test int: a parameter`
Another important prerequisite is that sphinx doc strings require an empty line between the doc string body and the parameter info.

	"""This is a doc string.
	:param test: this fails and isn't parsed correctly (i.e. printer literally)

	:param test: this succeeds and is marked up correctly.
	"""

sphinx-epytext has been considered but that doesn't handle the empty line case.
@nvaccessAuto nvaccessAuto added this to the 2019.3 milestone Jan 20, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants