WIP Build pydoctor API docs via Read the docs system.

[adiroiban]

This is just a proof of concept. Trying to see if I can hack read the docs system to get apidocs together with narrative docs.

The result can be tested at https://twisted--1422.org.readthedocs.build/en/1422/

The changes from here should be moved to separate PRs each fixing a single issue.

The GitHub actions builds for narrative docs and apidocs are still used to make it easy to debug errors as the log output is nicer than Read The Docs... and at the same time, even if there are warnings, you can still see the result on read the docs.

Drive by fixes:

• Use only bin/admin scripts to build the docs
• update pydoctor to latest version instead of using an unreleased version
• Fix the tox env to fail on docs build errors

[adiroiban]

OK. We got the API docs in Read The docs.

The build link is here https://readthedocs.org/projects/twisted/builds/12020401/

Total build time 665 seconds... so about 10 minutes
Building the docs was just 115 seconds...so about 2 minutes, but it took a long time to copy all the 18306 API doc files.

The HTML version is at https://twisted--1422.org.readthedocs.build/en/1422/

with API docs at https://twisted--1422.org.readthedocs.build/en/1422/api/

[mthuurne]

Total build time 665 seconds... so about 10 minutes
Building the docs was just 115 seconds...so about 2 minutes, but it took a long time to copy all the 18306 API doc files.

The reason Twisted has so many API doc files is that it passes --html-write-function-pages to pydoctor, which causes it to write a separate HTML file for each function. If this option is omitted, the number of output files drops from 17823 to 2375.

I don't know why separate function pages are being generated, since those pages are not linked from any other page and therefore only reachable if you jump directly to a function's exact URL.

I also don't know why my file count is slightly lower; did you include documentation files not written by pydoctor perhaps?

[adiroiban]

Thanks @mthuurne for your feedback.

It helped identify a big issue. Sphinx API links inside the narrative documentation are generated using absolute link.

---

The number was a quick and crude count which I found when I tried to remove the output 'apidocs' dir.

$ rm -rf apidocs/
Display all 18304 possibilities? (y or n)

---

As a test, I have updated the buildscript to not create the separate HTML files.

Will try to refactor that doc release code (ex merge APIBuilder and BuildAPIDocsScript ). I have already started but I didn't want to create a big diff, and just focus on the proof of concept for getting apidocs on Read The docs.

---

Without separate HTML files the build is much faster:

• Read The docs just creating the files - 66 seconds
• Total read the docs build including publishing - 374 seconds

For reference, without the API docs builds the Read The docs times are:

• Read The docs just creating the files - 16 seconds
• Total read the docs build including publishing - 139 sec

---

The separate HTML files change was introduced by exarkun here 828cb42

And was requested here https://twistedmatrix.com/trac/ticket/3241

The issue is "The howto docs generate links to functions."

I think that this is the way Sphinx API link builder works.

So... I think that we need a separate PR to see if we can fix the the apilinks extension to link to the class page...and maybe to a relative path.

I have updated the PR description with the current known issues

[adiroiban]

I think that we have a good start for the next release

The links from Sphinx narrative docs are generated using absolute URLs, and in this PR it was updated to use relative links.

For example, if you go to https://twisted--1422.org.readthedocs.build/en/1422/conch/howto/conch_client.html and then you click o the first link at may be to use twisted.conch.endpoints.SSHCommandClientEndpoint

The links from Sphinx narrative docs are generated using function HTML page, rather than anchors to class page.

For example if you go to https://twisted--1422.org.readthedocs.build/en/1422/core/howto/servers.html and then click on the link from or more information, or IReactorTCP.listenTCP and the other it will get you function page and not the class page.

[adiroiban]

This was moved to #1466