Use sphinx to create NVDA developer documentation

[LeonarddeR]

Link to issue number:

Fixes #9840

Summary of the issue:

When we switched to Python 3, we had to abandon Epydoc for developer documentation.

From the alternatives, sphinx seemed to be the most useful to us:

1. It is the only supported api doc generator on Read the Docs
2. It is used by numerous other projects, including python, wxPython, pyserial and configobj
3. It is heavily extensible.
4. In the end, it might allow us to integrate the user docs in the build process, thereby adding the ability to abandon txt2tags. Even if we don't, sphinx has markdown support with a simple extension.

Description of how this pull request fixes the issue:

This pr reimplements the functionality of scons devDocs, they are now build using sphinx.

Testing performed:

Tested that building the docs results into a structure of html files and that the build log doesn't contain any warnings related to parse errors of the source code.

Known issues with pull request:

• The layout of the dev docs is likely to be broken, as we are still using epidoc style doc strings that have to be converted to the Python (restructured text) style doc strings
• Read the Docs builds docs on linux. If we want to support this, it will require some source code changes. We can work around a lot of cases using mock imports. @derekriemer suggested that it might help if we build the docs ourselves on appveyor and push them afterwards. That's something that might be possible with RTD in the near future, but if I'm correct, it isn't possible yet.
• No build in support for type hints, though there is an extension for that.

Incompatible doc string syntax

Currently, we use epytext/epydoc style annotations for our doc strings. such as:

@param test: a parameter
@type test: int

Sphinx uses directives like :param and :type. It also allows nesting them, e.g. :param test int: a parameter
Another important prerequisite is that sphinx doc strings require an empty line between the doc string body and the parameter info.

"""This is a doc string.
:param test: this fails and isn't parsed correctly (i.e. printer literally)

:param test: this succeeds and is marked up correctly.
"""

I've looked that sphinx-epytext but that doesn't handle the empty line case. It also just replaces @ with : in most cases.

I found an interesting tutorial that deals with the conversion using sed. I myself chose a different approach and wrote a prototype python conversion script

@feerrenrut suggested using 2to3 for this. This has to be done in a follow up pr, though.

Change log entry:

• Changes for developers
  • Developer documentation is now build using sphinx. (Python 3: Restore developer documentation #9840)

[LeonarddeR]

I just looked at Sphinx AutoAPI which seems to offer a cleaner approach to build the docs by means of source code parsing. However, the logging of this tool is not very good. For example, if there is an error in a source file, the log file doesn't contain the name of the source file that actually failed. It also doesn't support surrogate characters in source files and seems to break on raw bytes of which it thinks they should UTF-8 continuation bytes.

[dpy013]

hi leonardder
Thank you for your work
This pr is a good start, which is a good thing for translators and maintainers.
Perhaps in the near future, user guides and update logs can be maintained with poedit.

[dpy013]

hi @michaelDCurran
What is the current progress of this pr, this pr is very important?
thank

[lukaszgo1]

Can epydoc be removed from misc-deps while at it?

[AppVeyorBot]

PR introduces Flake8 errors 😲

See test results for Failed build of commit 915df16915

[feerrenrut]

This doesn't look like it should be in this change.

[LeonarddeR]

You're correct, it makes sense to split this out.

[feerrenrut]

Or this?

[feerrenrut]

There seem to be some unrelated changes in this PR. I think the next step is to get this building in the scons devDocs command. Does it cause errors if the doc strings are not converted? I think focusing on the minimal changes to get this building without errors first. Then look at converting the docstrings or finding an appropriate plugin.

[feerrenrut]

I'd prefer to see these pieces happen as separate PR's though, if possible.

[dpy013]

I'd prefer to see these pieces happen as separate PR's though, if possible.

hi feerrenrut
We can split this pr, then review and merge.
thanks

[AppVeyorBot]

PR introduces Flake8 errors 😲

See test results for Failed build of commit bdedb60376

[AppVeyorBot]

PR introduces Flake8 errors 😲

See test results for Failed build of commit ae97234d63

[LeonarddeR]

I stripped this pr in such a way that it no longer contains changes to the NVDA source. All changes to NVDA's source that are required to make things work are no filed in separate pull requests: #10573, #10571 and #10574.

[feerrenrut]

Once #10573, #10571 and #10574 are merged, what is plan? I assume making sphinx docs build with the scons devDocs command. If that is happening without errors, then we can merge this in I think.

If you decide to go down the "roll your own" path for converting epydoc to sphinx you might want to consider using the 2to3 converter. It will handle iterating over the python files in a directory, and is smarter than regex about identifying python syntax. Modifying only doc strings should be much easier, if you hit that as a problem.

[LeonarddeR]

I assume making sphinx docs build with the scons devDocs command.

Yes, that's True.

If you decide to go down the "roll your own" path for converting epydoc to sphinx you might want to consider using the 2to3 converter. It will handle iterating over the python files in a directory, and is smarter than regex about identifying python syntax. Modifying only doc strings should be much easier, if you hit that as a problem.

That's an interesting one. I'd say we should do that in a separate pr though.

[dpy013]

hi @LeonarddeR I checked the files about the development documents uploaded by pr 19430 and found the following problems:

1. The uploaded file is not compressed, it is recommended to upload it to appveyor after compression
2. I hope that the development documents can have a format file of gettext.pot so that translators can localize the developer documents.
   thanks

[LeonarddeR]

hi @LeonarddeR I checked the files about the development documents uploaded by pr 19430 and found the following problems:

1. The uploaded file is not compressed, it is recommended to upload it to appveyor after compression

2. I hope that the development documents can have a format file of gettext.pot so that translators can localize the developer documents.
   thanks

Publishing devDocs in the output folder was a test to see whether the devDocs build on appveyor. I disabled this again as we never did this before, however I'm intending to revisit this at some point in a follow up.

[feerrenrut]

In appveyor yml we refer to "developerGuide" when doing a release. Having moved the developerGuide file and the release path being harder to test, there is potential for a change in behavior here. What do you think?

Could you update the testing to talk about the testing you have done with Appveyor?

[LeonarddeR]

I think that we should publish the changes, user guide and developer guide with every build, which makes it easier to inspect them.

[LeonarddeR]

Could you update the testing to talk about the testing you have done with Appveyor?

I have temporarily added the devDocs target to the appveyor scons targets and the build succeeded. Note that I only did this to see whether the dev docs would be able to build on appveyor. I don't think we should build them by default just yet.

[feerrenrut]

This looks ok, is there more work remaining before we merge this?

[LeonarddeR]

Nope, if this looks ok to you, go ahead and merge this. We'll have to fineutne the output anyway

[feerrenrut]

I have built this locally, bear in mind there are many warnings and a few errors. These will have to be fixed piece by piece, a single large PR will be hard to manage.
The following showed up a few times for the vision framework: TypeError: 'vision' object does not support item assignment

Full error:

WARNING: autodoc: failed to import module 'screenCurtain' from module 'visionEnhancementProviders'; the following exception was raised:
Traceback (most recent call last):
  File "C:\Users\Reef\AppData\Local\Programs\Python\Python37-32\lib\site-packages\sphinx\ext\autodoc\importer.py", line 32, in import_module
    return importlib.import_module(modname)
  File "C:\Users\Reef\AppData\Local\Programs\Python\Python37-32\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "C:\work\repo\nvda\6\source\visionEnhancementProviders\screenCurtain.py", line 292, in <module>
    class ScreenCurtainProvider(providerBase.VisionEnhancementProvider):
  File "C:\work\repo\nvda\6\source\visionEnhancementProviders\screenCurtain.py", line 293, in ScreenCurtainProvider
    _settings = ScreenCurtainSettings()
  File "C:\work\repo\nvda\6\source\vision\providerBase.py", line 42, in __init__
    self.initSettings()  # ensure that settings are loaded at construction time.
  File "C:\work\repo\nvda\6\source\autoSettingsUtils\autoSettings.py", line 107, in initSettings
    self._initSpecificSettings(self, self.supportedSettings)
  File "C:\work\repo\nvda\6\source\autoSettingsUtils\autoSettings.py", line 89, in _initSpecificSettings
    config.conf[section][settingsId] = {}
TypeError: 'vision' object does not support item assignment