Support for type hints in function signatures

[wsanchez]

If one has type hints in code, adding @type and @rtype info in doc strings is redundant.

[mthuurne]

In many cases, type annotations can indeed replace @type and @rtype. But I wonder how far pydoctor should go in this. For example:

class C:
    a: int
    b = True
    c = 1 + 1 + 1
    d = ':'.join(['x', 'y', 'z'])

In this example:

• the type of a is explicit in the annotation
• the type of b is implicit, but can be trivially inferred
• the types of c and d are implicit and while they can be inferred, this is not trivial

As you can see in the example, even in type annotated code, types aren't always trivial. I don't think pydoctor should attempt to do full type inference, since that is a whole complex project of its own.

Ideally, we'd be able to get type inference from a library, but I'm not aware of a good library that does this. The most advanced type inference for Python I've seen is in mypy, but mypy doesn't have a library or service interface at the moment that fits pydoctor's needs. The Astroid library used by PyLint should be easier to integrate, but it doesn't have PEP 484 support yet, so it doesn't fit our needs either, at least not in the near future.

Still, even if we support only explicit and trivially inferred types, that would already avoid a lot of duplicate type information, so I think that's something we should absolutely do.

[mthuurne]

Another thing to consider is whether we want to support type hints when running on Python versions before 3.6, or support the use of # type: comments instead of only annotations. If so, we should probably use the typed_ast library.

Note that the typed_ast README states it will not be updated for Python 3.8, which means that typed_ast cannot replace ast completely, only replace it under certain conditions.

Personally, I'm not in favor of supporting type hints on Python versions below 3.6. It would require additional effort that I think is better spent elsewhere, not only to implement it but also in having to maintain and test support for two parsers. Additionally, I think it would be very useful to adopt type annotations in the pydoctor source itself, which would require pydoctor to run on Python 3.6 or higher, eliminating one of the advantages typed_ast could bring.

[wsanchez]

Support for 2.7-compatible type hints is clearly not worth the bother.

I don't think that requiring pydoctor to run on Py3.6 or higher is a minus at this point.

pydoctor needn't take on type inference here. Right now, it requires @ivar or whatever explicit annotations in the doc strings, so requiring that those annotations more to explicit type hints is not a functional regression. If we find a library we can use for type inference, then we can start doing that, but that's an improvement for later.

[mthuurne]

I'm working on an implementation of type hints support that uses annotations from nodes produced by the ast parser. This means only projects using the 3.5/3.6 annotation syntax and running pydoctor under Python 3 can take advantage of it, but for projects targeting an older Python version there is still the @type docstring tag.

Since #170 @var/@cvar/@ivar are no longer the only way to document a variable: you can also put a docstring describing one variable below an assignment. Type hints are a natural fit for this syntax:

class C:
    def __init__(self, secret: int) -> None:

        self.secret = secret
        """Code to open my luggage."""

        self.merchandise: List[str] = []
        """Where the real money from the movie is made."""

In the latest release the mypy daemon got experimental support for returning inferred types for functions. So this might be a way to get type inference in the future.

[moshez]

Based on https://github.com/twisted/pydoctor/pull/175/files#diff-1e6d0f0013af649be87039388d3b105dR682 I think this ticket can be marked "closed"?

Or is there something that still needs to be done (in which case I would like to understand what)?

[mthuurne]

The main thing that is still missing is support for annotations and type comments on function signatures. The PR you linked implements support for variables only.

I had a look at implementing it for functions, but the code isn't structured well for that: we'd have to merge type information discovered at different times (when processing AST nodes and when parsing docstrings), but the FieldHandler data structure isn't designed for that.

I'd prefer to get a release out before doing major refactoring, for multiple reasons:

• to have more time to catch regressions before the release after that
• so we can drop Python 2.7 support and add type hints to pydoctor itself, so refactoring can be done with help from mypy

While the branch coverage of our unit tests is starting to approach acceptable levels, this may be deceptive, since a lot of data structures are used in ad-hoc ways and the unit tests don't cover a whole lot of state space. I'd feel a lot more confident making invasive changes if I could use mypy in addition to unit and system tests.

In theory mypy can run on Python 2.7 code, but the type comments syntax is just too painful to work with in my opinion, as I'm used to Python 3.6 syntax now.

[moshez]

Thanks for the explanation! This makes sense. Two questions:

[1] Can we retitle this issue to "support for type hints in function signatures" to make it clearer?
[2] What needs to be done to release? The release process itself seems straightforward, so I'm wondering if it's pending on any other tickets?

[mthuurne]

PR #199 and #200 are ready to be reviewed; I'd like to get those in before a release.

The README could use an update, since I think pydoctor is useful for projects that are looking for a successor to Epydoc, so its potential audience is not only Twisted. People from other projects have already inquired about a release, see #96.

I also think that every release should include some kind of release notes or change log, even if it's minimal. It seems pydoctor didn't do this in the past, but if we want to broaden its use beyond Twisted, we should communicate to users why they'd want to update.

All in all, it shouldn't be a lot of work, but I can't review my own PRs, so I will need some help. @twm and @glyph together reviewed seven PRs this week, so I'm not complaining, but I'd really like to get the two remaining ones in as well.

[mthuurne]

PR #206 updates docs and metadata for a release. Please review. Thanks to @twm for the review.

Only PR #200 is remaining now. I'd really like to get that one in, since it fixes errors that are visible in the output. It also eliminates the last errors that the twisted-apidocs tox env produces and releasing with a fully green CI status would be a good milestone.

[moshez]

Posted a review. Let me know if there are any other blockers (for example, if you do decide to split the PR, or need any reviews for the release process! Feel free to e-mail me directly.

[mthuurne]

The next release of pydoctor will have support for type annotations on functions.

There is some simple type inference done for variables. If this is insufficient, users can work around that by explicitly adding annotations or type docstring fields. I still like to idea of using mypyd from a technical point of view, but at the moment I think there are other things to work on that yield more results.

So I'm closing this issue now. If we ever want to improve type inference in the future, that can be tracked in a new issue.

[glyph]

The next release of pydoctor will have support for type annotations on functions.

Amazing. This is fantastic news. How long until the next release? :)

[mthuurne]

How long until the next release? :)

A few days at most. See #290 for details.