#10066 Update pydoctor version and remove Twisted pydoctor custom code that is now available as standard features. #11578

adiroiban · 2022-07-12T15:03:11Z

Scope and purpose

If the epydoc API markup is invalid, the CI is happy and green.

We need to update the CI to fail on pydoctor build errors.
This should help make sure the API doc is valid and sane :)

As part of this PR I have updated the pydoctor version.

Also, instead of using the custom python script to call pydoctor, this calls pydoctor directly from tox.
I hope that in this way it should be easier to understand what is going on there, with fewer redirection layers.

Contributor Checklist:

I ran tox -e lint to format my patch to meet the Twisted Coding Standard
I have created a newsfragment in src/twisted/newsfragments/ (see: News files)
The title of the PR starts with the associated issue number (without the # character).
I have updated the automated tests and checked that all checks for the PR are green.
The merge commit will use the below format
The first line is automatically generated by GitHub based on PR ID and branch name.
The other lines generated by GitHub should be replaced.

Merge pull request #123 from twisted/4356-branch-name-with-issue-id

Author: adiroiban
Reviewer: 
Fixes #10066

Update pydoctor version.
Fail CI for a PR on pydoctor build errors.

…cs errors.

tristanlatr · 2022-07-20T12:39:33Z

tox.ini


 extras = dev_release
-commands = {toxinidir}/bin/admin/build-apidocs {toxinidir}/src/ apidocs
+commands =
+    pydoctor --quiet \


This is missing --template-dir option.

setup.cfg

adiroiban · 2022-07-20T13:09:34Z

Many thanks for the review.

I will try to reduce the things changed in this PR.

For now, I just want to update the pydoctor version and see that CI fails on pydoctor errors.

In a separate ticket we can look at fixing the pydoctor errors and also cleaning the things that are no longer needed.

I will see how hard is to fix the current pydoctor errors.
I would like to fix them in this PR, so that we can have a green CI run.

The extra cleanup can be done in a separate PR.

tristanlatr · 2022-07-20T13:15:54Z

The build is currently failing with error: could not import module twisted.python._pydoctor.

I'de suggest to simply drop the custom twisted.python._pydoctor.TwistedSystem because it's not compatible with pydoctor 22.7 anyway. Then, in the later PR, adjust it and we can re-enable it (it will only contains the inventory tweaks at this point).

adiroiban · 2022-07-20T13:26:15Z

thanks @tristanlatr

I have updated twisted/twisted-contributors team. Not sure why you weren't there.

If you have time. Can you make the required changes to get pydoctor working ?

Thanks

tristanlatr · 2022-07-20T13:29:35Z

Alright, but there is one thing that bother me. It’s like the _pydoctor module is not included in the default dist build. So when we’ll need to re enable the custom system again, this module needs to be installed. You know what I mean ?

adiroiban · 2022-07-20T16:43:36Z

It’s like the _pydoctor module is not included in the default dist build.

I don't know why it is not included.
But I think that is a bug.

In the code, I see that the intention is to have it included.

twisted/MANIFEST.in

Lines 40 to 44 in fc20692

    
           # Include our docs templates 
        
           recursive-include src/twisted/python/_pydoctortemplates *.html 
        
           # Include all modules, even on a Python we're not installing for 
        
           recursive-include src/twisted *.py

python setup.py bdist_wheel

adding 'twisted/python/_pydoctor.py'

and it was installed in my environment

tristanlatr · 2022-07-20T17:27:51Z

Ok, maybe the twisted package is not correctly installed in the tox environment then.

for more information, see https://pre-commit.ci

…ed into 10066-pydoctor-and-docs-ci

tristanlatr · 2022-07-28T03:41:48Z

The pydoctor build is fixed, I believe. Though, I haven't read the logs in the CI, but tox -e apidocs runs on my local device.

This leaves the warnings to fix:

/Users/tristan/Documents/twisted/src/twisted/test/test_adbapi.py:362: Assignment to constant "DB_DIR" inside an instance is ignored, this value will not be part of the docs.
/Users/tristan/Documents/twisted/src/twisted/trial/test/test_assertions.py:1610: Invalid call to twisted.python.deprecate.deprecated(), first argument should be a call to incremental.Version()
/Users/tristan/Documents/twisted/src/twisted/trial/test/test_assertions.py:1618: Invalid call to twisted.python.deprecate.deprecated(), first argument should be a call to incremental.Version()
/Users/tristan/Documents/twisted/src/twisted/trial/_dist/functional.py:34: bad docstring: Improper paragraph indentation.
/Users/tristan/Documents/twisted/src/twisted/web/iweb.py:674: Cannot find link target for "__init__"
/Users/tristan/Documents/twisted/src/twisted/web/iweb.py:686: Cannot find link target for "__init__"
/Users/tristan/Documents/twisted/src/twisted/conch/checkers.py:117: Cannot find link target for "twisted.conch.checkers.pwd.struct_passwd", resolved from "pwd.struct_passwd"
/Users/tristan/Documents/twisted/src/twisted/conch/checkers.py:136: Cannot find link target for "spwd.struct_spwd" (you can link to external docs with --intersphinx)
/Users/tristan/Documents/twisted/src/twisted/conch/checkers.py:41: Cannot find link target for "twisted.conch.checkers.pwd.struct_passwd", resolved from "pwd.struct_passwd"
/Users/tristan/Documents/twisted/src/twisted/conch/checkers.py:86: Cannot find link target for "twisted.conch.checkers.pwd.struct_passwd", resolved from "pwd.struct_passwd"
/Users/tristan/Documents/twisted/src/twisted/conch/checkers.py:86: Cannot find link target for "spwd.struct_spwd" (you can link to external docs with --intersphinx)
/Users/tristan/Documents/twisted/src/twisted/internet/tcp.py:918: Cannot find link target for "errno.WSAEMFILE", resolved from "EMFILE" (you can link to external docs with --intersphinx)
/Users/tristan/Documents/twisted/src/twisted/logger/_logger.py:179: Cannot find link target for "Failure"
/Users/tristan/Documents/twisted/src/twisted/spread/jelly.py:211: Cannot find link target for "twisted.spread.flavors.setUnjellyableForClass", resolved from "setUnjellyableForClass"
/Users/tristan/Documents/twisted/src/twisted/web/_flatten.py:454: Cannot find link target for "Deferred"
/Users/tristan/Documents/twisted/src/twisted/web/_flatten.py:462: Cannot find link target for "Deferred"
/Users/tristan/Documents/twisted/src/twisted/web/_flatten.py:477: Cannot find link target for "Deferred"
/Users/tristan/Documents/twisted/src/twisted/web/_template_util.py:1069: Cannot find link target for "IRequest"
/Users/tristan/Documents/twisted/src/twisted/web/_template_util.py:1070: Cannot find link target for "IRenderable"
these 2 objects' docstrings contain syntax errors:
    twisted.python._pydoctor
    twisted.trial._dist.functional.takeWhile

Let's note that pydoctor has this known bug: twisted/pydoctor#295 which might explain some of the errors.

Also: twisted.python.deprecate warnings are still triggered for twisted/trial/test/test_assertions.py:1610 even if the module is hidden because the warning comes from the AST parsing step, which is done for all source, no matter is there's hidden, unlike the docstring parsing.

adiroiban · 2022-07-28T17:39:52Z

Ok, One test failing

Traceback (most recent call last):
  File "/home/runner/work/twisted/twisted/.tox/alldeps-withcov-posix/lib/python3.8/site-packages/twisted/trial/runner.py", line 688, in loadByName
    return self.suiteFactory([self.findByName(name, recurse=recurse)])
  File "/home/runner/work/twisted/twisted/.tox/alldeps-withcov-posix/lib/python3.8/site-packages/twisted/trial/runner.py", line 451, in findByName
    obj = reflect.namedModule(searchName)
  File "/home/runner/work/twisted/twisted/.tox/alldeps-withcov-posix/lib/python3.8/site-packages/twisted/python/reflect.py", line 156, in namedModule
    topLevel = __import__(name)
  File "/home/runner/work/twisted/twisted/.tox/alldeps-withcov-posix/lib/python3.8/site-packages/twisted/python/test/test_pydoctor.py", line 18, in <module>
    from twisted.python._pydoctor import TwistedSphinxInventory, TwistedSystem
  File "/home/runner/work/twisted/twisted/.tox/alldeps-withcov-posix/lib/python3.8/site-packages/twisted/python/_pydoctor.py", line 17, in <module>
    from pydoctor import astbuilder, model, zopeinterface  # type: ignore[import]
builtins.ImportError: cannot import name 'zopeinterface' from 'pydoctor' (/home/runner/work/twisted/twisted/.tox/alldeps-withcov-posix/lib/python3.8/site-packages/pydoctor/__init__.py)

twisted.python.test.test_pydoctor

tristanlatr · 2022-07-28T17:57:40Z

Yes, this is normal, pydoctor 22.7 introduced a breaking change in this regards, but we never had a stable API, that was clear from the beginning.

I'll just cleanup _pydoctor.py is that OK with you.

adiroiban · 2022-07-28T18:05:09Z

I'll just cleanup _pydoctor.py is that OK with you.

Yes. Thanks.

Feel free to push anything.

I can then see how to break this PR to get these changes reviewed and merged.

Newer versions of pydoctor includes support for twisted.python.deprecate and allows to tweak the privacy of some objects.

for more information, see https://pre-commit.ci

tristanlatr · 2022-07-28T18:39:28Z

The tests are now all successful. The only thing that bothers me is that we’re not actually using the custom system, we just have tests for it.

And more, I don’t think it’s required anymore, meaning zope has fixed it’s wrong inventory links: because I can’t see any warnings that refers to the zope api. So I believe we can trash it all together.

adiroiban · 2022-07-28T20:29:32Z

The only thing that bothers me is that we’re not actually using the custom system, we just have tests for it.

We can remove the custom system code if not used.

less is more :)

for more information, see https://pre-commit.ci

…ed into 10066-pydoctor-and-docs-ci

tristanlatr · 2022-07-28T21:07:53Z

Ok, I believe it's pretty clean now.

tristanlatr · 2022-07-28T21:08:49Z

I've enabled the readthedocs theme, I believe that was the plan :) But tell me if I should use the classic theme

adiroiban · 2022-07-29T00:39:24Z

These changes are awesome.

I love the config file.

The theme is also ok.

Just some feedback for the theme.

There are now 2 main links:

Twisted main site
API docs index

Maybe instead of Twisted main site the link should be to the narrative docs base - like https://twisted--11578.org.readthedocs.build/en/11578/index.html this should match RTD links.

Now, in order to have this merged ASAP, as you mentioned on the issue, I think that is best to reduce the scope of this PR.

Instead of pydoctor upgrade + ci fail, we should use this PR only to upgrade the pydoctor version.

I/we can look to fix the pydoctor errors in a separate PR.

glyph

LGTM. Love to see all this unnecessary support code getting zapped.

glyph · 2022-07-29T00:38:43Z

docs/conf.py

-    "--project-name=Twisted",
-    "--project-url=https://twistedmatrix.com/",
-    "--system-class=twisted.python._pydoctor.TwistedSystem",
-    "--docformat=epytext",
-    "--intersphinx=https://docs.python.org/3/objects.inv",
-    "--intersphinx=https://cryptography.io/en/latest/objects.inv",
-    "--intersphinx=https://pyopenssl.readthedocs.io/en/stable/objects.inv",
-    "--intersphinx=https://hyperlink.readthedocs.io/en/stable/objects.inv",
-    "--intersphinx=https://twisted.org/constantly/docs/objects.inv",
-    "--intersphinx=https://twisted.org/incremental/docs/objects.inv",
-    "--intersphinx=https://python-hyper.org/projects/hyper-h2/en/stable/objects.inv",
-    "--intersphinx=https://priority.readthedocs.io/en/stable/objects.inv",
-    "--intersphinx=https://zopeinterface.readthedocs.io/en/latest/objects.inv",
-    "--intersphinx=https://automat.readthedocs.io/en/latest/objects.inv",


yay for getting this to be declarative data in setup.cfg and not stuffed into command-line args!

glyph · 2022-07-29T00:40:40Z

bin/admin/build-apidocs

@@ -1,19 +0,0 @@
-#!/usr/bin/env python


Love to see all the ./build/admin scripts get replaced with tox environments. Much more discoverable and less error-prone.

glyph · 2022-07-29T00:42:26Z

setup.cfg

+    HIDDEN:twisted.words.test
+    HIDDEN:twisted.web.test
+    HIDDEN:twisted.spread.test
+    HIDDEN:twisted.scripts.test
+    HIDDEN:twisted.runner.test
+    HIDDEN:twisted.python.test
+    HIDDEN:twisted.protocols.haproxy.test
+    HIDDEN:twisted.protocols.test
+    HIDDEN:twisted.positioning.test
+    HIDDEN:twisted.persisted.test
+    HIDDEN:twisted.pair.test
+    HIDDEN:twisted.names.test
+    HIDDEN:twisted.mail.test
+    HIDDEN:twisted.logger.test
+    HIDDEN:twisted.cred.test
+    HIDDEN:twisted.conch.test
+    HIDDEN:twisted.application.runner.test
+    HIDDEN:twisted.application.twist.test
+    HIDDEN:twisted.application.test
+    HIDDEN:twisted._threads.test
+    HIDDEN:twisted.trial._dist.test
+    HIDDEN:twisted.trial.test
+    HIDDEN:twisted.internet.test
+    HIDDEN:twisted.test.*
+    PUBLIC:twisted.test.proto_helpers


Beautiful. Self-documenting!

glyph · 2022-07-29T01:28:43Z

Auto-merge set so we can land this as soon as static check and api doc build issues are corrected. (Looks like maybe an intersphinx thing for the docs build?)

tristanlatr · 2022-07-29T01:44:33Z

Looks like maybe an intersphinx thing for the docs build?

No, it looks like months of not checking the pydoctor warnings ;-)

should we fix the docstring links in this PR?

adiroiban · 2022-07-29T09:27:26Z

Ok. The plan is to land this PR asap.

The remaining pydoctor "compilation" error should be fixed as a followup in #11590

Those errors already exist in trunk, so they are not errors introduced in this PR.

Is just that with this PR, pydoctor is detecting more errors.

adiroiban added 2 commits July 12, 2022 15:57

Upgrade pydoctor. Create dedicated apidoc builder that fails on apido…

66c902e

…cs errors.

Remove custom apidocs builder script and tests.

376b1b6

adiroiban changed the title ~~10066 Update pydoctor and fail CI on pydoctor errors~~ 10066 Update pydoctor and fail CI on pydoctor errors. Jul 12, 2022

adiroiban mentioned this pull request Jul 12, 2022

Update pydoctor version and remove Twisted pydoctor custom code that is now available as standard features. #10066

Closed

adiroiban added 2 commits July 12, 2022 16:10

Fix python setup.

bce46d3

Run all docs in a separate job to make it easier to check the output.

e19c4b5

This was referenced Jul 15, 2022

Move release related code outside of the twisted package #3138

Closed

#11585 Update the PR template and dev docs to not require a specific merge commit message. #11586

Merged

tristanlatr reviewed Jul 20, 2022

View reviewed changes

adiroiban added 2 commits July 20, 2022 14:03

Merge branch 'trunk' into 10066-pydoctor-and-docs-ci

7676e28

Update after initial review.

f94c083

tristanlatr and others added 8 commits July 27, 2022 21:53

Drop custom pydoctor system.

de914fa

Hide tests

6e37ad2

[pre-commit.ci] auto fixes from pre-commit.com hooks

fa09246

for more information, see https://pre-commit.ci

Configure privacy tweaks

3470baa

Merge branch '10066-pydoctor-and-docs-ci' of github.com:twisted/twist…

b90cfde

…ed into 10066-pydoctor-and-docs-ci

Move all template customization in a one place

f3d4122

Fix typo

dc25161

Fix paragraph indentation in _pydoctor.py

ba872b1

adiroiban and others added 2 commits July 28, 2022 10:15

Merge branch 'trunk' into 10066-pydoctor-and-docs-ci

e096fa1

Fix ci.

cc52861

adiroiban force-pushed the 10066-pydoctor-and-docs-ci branch from 2942f22 to cc52861 Compare July 28, 2022 10:43

tristanlatr and others added 2 commits July 28, 2022 14:06

Cleanup _pydoctor.py and test_pydoctor.py.

3727e64

Newer versions of pydoctor includes support for twisted.python.deprecate and allows to tweak the privacy of some objects.

[pre-commit.ci] auto fixes from pre-commit.com hooks

a2f45b5

for more information, see https://pre-commit.ci

Use config file not to duplicate pydoctor CLI args

7e8fd98

pre-commit-ci bot and others added 3 commits July 28, 2022 20:32

[pre-commit.ci] auto fixes from pre-commit.com hooks

c4c6a4a

for more information, see https://pre-commit.ci

Completely drop custom pydoctor system.

2405d03

Merge branch '10066-pydoctor-and-docs-ci' of github.com:twisted/twist…

70a2883

…ed into 10066-pydoctor-and-docs-ci

Use readthedocs theme

faa73f1

adiroiban changed the title ~~10066 Update pydoctor and fail CI on pydoctor errors.~~ #10066 Update pydoctor and fail CI on pydoctor errors. Jul 29, 2022

glyph approved these changes Jul 29, 2022

View reviewed changes

chevah-robot added the needs-merge label Jul 29, 2022

chevah-robot assigned adiroiban Jul 29, 2022

glyph enabled auto-merge July 29, 2022 01:27

adiroiban changed the title ~~#10066 Update pydoctor and fail CI on pydoctor errors.~~ #10066 Update pydoctor version and remove Twisted pydoctor custom code that is now available as standard features. Jul 29, 2022

adiroiban mentioned this pull request Jul 29, 2022

Fix pydoctor errors generated after the pydoctor upgrade #11590

Open

Ignore pydoctor error for now. They will be fixed in #11590

1c7ebf4

glyph merged commit ff2ea61 into trunk Jul 29, 2022

glyph deleted the 10066-pydoctor-and-docs-ci branch July 29, 2022 09:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

#10066 Update pydoctor version and remove Twisted pydoctor custom code that is now available as standard features. #11578

#10066 Update pydoctor version and remove Twisted pydoctor custom code that is now available as standard features. #11578

adiroiban commented Jul 12, 2022

tristanlatr Jul 20, 2022

adiroiban commented Jul 20, 2022

tristanlatr commented Jul 20, 2022

adiroiban commented Jul 20, 2022

tristanlatr commented Jul 20, 2022

adiroiban commented Jul 20, 2022

tristanlatr commented Jul 20, 2022

tristanlatr commented Jul 28, 2022 •

edited

adiroiban commented Jul 28, 2022

tristanlatr commented Jul 28, 2022

adiroiban commented Jul 28, 2022

tristanlatr commented Jul 28, 2022 •

edited

adiroiban commented Jul 28, 2022

tristanlatr commented Jul 28, 2022

tristanlatr commented Jul 28, 2022 •

edited

adiroiban commented Jul 29, 2022

glyph left a comment

glyph Jul 29, 2022

tristanlatr Jul 29, 2022

glyph Jul 29, 2022

glyph Jul 29, 2022

glyph commented Jul 29, 2022

tristanlatr commented Jul 29, 2022

adiroiban commented Jul 29, 2022

#10066 Update pydoctor version and remove Twisted pydoctor custom code that is now available as standard features. #11578

#10066 Update pydoctor version and remove Twisted pydoctor custom code that is now available as standard features. #11578

Conversation

adiroiban commented Jul 12, 2022

Scope and purpose

Contributor Checklist:

tristanlatr Jul 20, 2022

Choose a reason for hiding this comment

adiroiban commented Jul 20, 2022

tristanlatr commented Jul 20, 2022

adiroiban commented Jul 20, 2022

tristanlatr commented Jul 20, 2022

adiroiban commented Jul 20, 2022

tristanlatr commented Jul 20, 2022

tristanlatr commented Jul 28, 2022 • edited

adiroiban commented Jul 28, 2022

tristanlatr commented Jul 28, 2022

adiroiban commented Jul 28, 2022

tristanlatr commented Jul 28, 2022 • edited

adiroiban commented Jul 28, 2022

tristanlatr commented Jul 28, 2022

tristanlatr commented Jul 28, 2022 • edited

adiroiban commented Jul 29, 2022

glyph left a comment

Choose a reason for hiding this comment

glyph Jul 29, 2022

Choose a reason for hiding this comment

tristanlatr Jul 29, 2022

Choose a reason for hiding this comment

glyph Jul 29, 2022

Choose a reason for hiding this comment

glyph Jul 29, 2022

Choose a reason for hiding this comment

glyph commented Jul 29, 2022

tristanlatr commented Jul 29, 2022

adiroiban commented Jul 29, 2022

tristanlatr commented Jul 28, 2022 •

edited

tristanlatr commented Jul 28, 2022 •

edited

tristanlatr commented Jul 28, 2022 •

edited