New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stubgen improvements: preserving existing annotations #3169

Merged
merged 22 commits into from Sep 22, 2017

Conversation

Projects
None yet
5 participants
@dmoisset
Contributor

dmoisset commented Apr 13, 2017

This PR modifies heavily the stubgen to be able to create better stubs when processing already annotated files (Implements #2106 ). It essentially adds annotations on the stubs for:

  • Variables/attributes with annotations
  • Annotations on functions/methods
  • TypeVar declarations
  • Type Aliases
    It also tries to add all the required imports into the stub to make the resulting file valid, and handle properly forward references when necessary. It also adds some fixes (handling of generics as base classes, some formatting fixes in the output) that were natural consequences of some of the refactoring I made.

I know this is a large patch (and refactors some pieces of the generator), so feel free to comment if it needs discussion.

@dmoisset

This comment has been minimized.

Show comment
Hide comment
@dmoisset

dmoisset Apr 13, 2017

Contributor

Just checked other stubgen issues and this seems to fix #1782 too

Contributor

dmoisset commented Apr 13, 2017

Just checked other stubgen issues and this seems to fix #1782 too

@dmoisset dmoisset changed the title from Stubgen imrpovements: preserving existing annotations to Stubgen improvements: preserving existing annotations Apr 13, 2017

@ilevkivskyi

I like this PR, it will definitely make stubgen more usable. Here are more detailed comments.

Show outdated Hide outdated test-data/unit/stubgen.test
y: Any
[case testMultipleAssignmentAnnotated]
x, y = 1, "2" # type: int, str

This comment has been minimized.

@ilevkivskyi

ilevkivskyi Apr 27, 2017

Collaborator

I would add few tests with wrong annotations, for example with more types than variables, and other possible mismatches, just to be sure that stubgen will not crash on those.

@ilevkivskyi

ilevkivskyi Apr 27, 2017

Collaborator

I would add few tests with wrong annotations, for example with more types than variables, and other possible mismatches, just to be sure that stubgen will not crash on those.

This comment has been minimized.

@dmoisset

dmoisset Apr 27, 2017

Contributor

OK... the current code is not crashing, but generating "something" instead of erroring out. I can put an error message although I'm a bit converned that validation is a bit out of scope from the stubgen tool.

@dmoisset

dmoisset Apr 27, 2017

Contributor

OK... the current code is not crashing, but generating "something" instead of erroring out. I can put an error message although I'm a bit converned that validation is a bit out of scope from the stubgen tool.

This comment has been minimized.

@ilevkivskyi

ilevkivskyi Apr 27, 2017

Collaborator

I'm a bit converned that validation is a bit out of scope from the stubgen tool.

Just not crashing on invalid input is already OK, I think. If you can produce a reasonable warning, then it's perfect.

@ilevkivskyi

ilevkivskyi Apr 27, 2017

Collaborator

I'm a bit converned that validation is a bit out of scope from the stubgen tool.

Just not crashing on invalid input is already OK, I think. If you can produce a reasonable warning, then it's perfect.

This comment has been minimized.

@dmoisset

dmoisset Sep 1, 2017

Contributor

OK. the current status is that it doesn't crash, just ignores the "extra" data

@dmoisset

dmoisset Sep 1, 2017

Contributor

OK. the current status is that it doesn't crash, just ignores the "extra" data

def f(self): ...
def g(self): ...
[case testExportViaRelativeImport]
from .api import get
[out]
from .api import get as get

This comment has been minimized.

@ilevkivskyi

ilevkivskyi Apr 27, 2017

Collaborator

Note that this form should be preserved. There is currently rule in PEP 484 that says that such form should be used for names that are actually imported (and therefore re-exported by Python runtime). Mypy currently does not follow this rule and treats all names as re-exported, see #2927, but it will be fixed soon.

Therefore, to prevent breakage of stubgen stubs after the fix, and to comply with PEP 484 this form from mod import X as X should be preserved for names that are actually imported in original file. At the same time, a short form from mod import X should be used for auxiliary/fake imports created by stubgen (like typing).

@ilevkivskyi

ilevkivskyi Apr 27, 2017

Collaborator

Note that this form should be preserved. There is currently rule in PEP 484 that says that such form should be used for names that are actually imported (and therefore re-exported by Python runtime). Mypy currently does not follow this rule and treats all names as re-exported, see #2927, but it will be fixed soon.

Therefore, to prevent breakage of stubgen stubs after the fix, and to comply with PEP 484 this form from mod import X as X should be preserved for names that are actually imported in original file. At the same time, a short form from mod import X should be used for auxiliary/fake imports created by stubgen (like typing).

This comment has been minimized.

@dmoisset

dmoisset Sep 1, 2017

Contributor

Even after reading this comment and pep-484, I'm not sure I fully understand what is required here. What I have is:

  • If a stub needs to export an imported name, it should import it with an alias (even if the alias is name is the same as the original). Is that correct?
  • But then, how does stubgen know which imported names where "intended" to be reexported? should it use __all__ and ignore everything else? should it reexport everything just in case? should it reexport only names imported in a certain way?

I could use a hand here at least with the test cases, once I have some boundaries defined by those I could probably tweak the implementation to match the spec.

@dmoisset

dmoisset Sep 1, 2017

Contributor

Even after reading this comment and pep-484, I'm not sure I fully understand what is required here. What I have is:

  • If a stub needs to export an imported name, it should import it with an alias (even if the alias is name is the same as the original). Is that correct?
  • But then, how does stubgen know which imported names where "intended" to be reexported? should it use __all__ and ignore everything else? should it reexport everything just in case? should it reexport only names imported in a certain way?

I could use a hand here at least with the test cases, once I have some boundaries defined by those I could probably tweak the implementation to match the spec.

This comment has been minimized.

@ilevkivskyi

ilevkivskyi Sep 1, 2017

Collaborator

If a stub needs to export an imported name, it should import it with an alias (even if the alias is name is the same as the original). Is that correct?

Yes.

But then, how does stubgen know which imported names where "intended" to be reexported? should it use all and ignore everything else? should it reexport everything just in case? should it reexport only names imported in a certain way?

I think the safest bet would be to re-export all globally visible names (i.e. ignore imports in function bodies), if there is not __all__ defined, and only re-export names form __all__ if there is one.

@ilevkivskyi

ilevkivskyi Sep 1, 2017

Collaborator

If a stub needs to export an imported name, it should import it with an alias (even if the alias is name is the same as the original). Is that correct?

Yes.

But then, how does stubgen know which imported names where "intended" to be reexported? should it use all and ignore everything else? should it reexport everything just in case? should it reexport only names imported in a certain way?

I think the safest bet would be to re-export all globally visible names (i.e. ignore imports in function bodies), if there is not __all__ defined, and only re-export names form __all__ if there is one.

This comment has been minimized.

@dmoisset

dmoisset Sep 22, 2017

Contributor

I'm implementing the __all__ scenario. I'm not doing the "reexport everything if there's no __all__" because I think the result ends up being unnecessarily noisy for most cases (you'll end up with tons of modules saying from typing import List as List because they don't have an __all__). In any case, it's easy to change our minds later

@dmoisset

dmoisset Sep 22, 2017

Contributor

I'm implementing the __all__ scenario. I'm not doing the "reexport everything if there's no __all__" because I think the result ends up being unnecessarily noisy for most cases (you'll end up with tons of modules saying from typing import List as List because they don't have an __all__). In any case, it's easy to change our minds later

This comment has been minimized.

@ilevkivskyi

ilevkivskyi Sep 22, 2017

Collaborator

you'll end up with tons of modules saying from typing import List as List because they don't have an __all__.

You can just manually exclude names imported from typing. In general, this is an important point, since it already caused some bugs in typeshed, see e.g. python/typeshed#1484. I would prefer this test to work.

(Also I don't think people will be unhappy about a bit more bulky stub, but they will be certainly unhappy about false positives when as name isn't present).

@ilevkivskyi

ilevkivskyi Sep 22, 2017

Collaborator

you'll end up with tons of modules saying from typing import List as List because they don't have an __all__.

You can just manually exclude names imported from typing. In general, this is an important point, since it already caused some bugs in typeshed, see e.g. python/typeshed#1484. I would prefer this test to work.

(Also I don't think people will be unhappy about a bit more bulky stub, but they will be certainly unhappy about false positives when as name isn't present).

def visit_unbound_type(self, t: UnboundType)-> str:
s = t.name
base = s.split('.')[0]
self.stubgen.import_tracker.require_name(base)

This comment has been minimized.

@ilevkivskyi

ilevkivskyi Apr 27, 2017

Collaborator

I am not sure I follow the logic here. If I have an unbound type X.Y.Z, will this require import X? But how do we know X is actually a module? Maybe it would be better to just use Any for unbound types?

@ilevkivskyi

ilevkivskyi Apr 27, 2017

Collaborator

I am not sure I follow the logic here. If I have an unbound type X.Y.Z, will this require import X? But how do we know X is actually a module? Maybe it would be better to just use Any for unbound types?

This comment has been minimized.

@dmoisset

dmoisset Sep 1, 2017

Contributor

when you have an X.Y.Z annotatation, what other things could X be? I'm assuming module here because if it's not the only choice, it's probably the best one.

@dmoisset

dmoisset Sep 1, 2017

Contributor

when you have an X.Y.Z annotatation, what other things could X be? I'm assuming module here because if it's not the only choice, it's probably the best one.

This comment has been minimized.

@ilevkivskyi

ilevkivskyi Sep 1, 2017

Collaborator

I am still not sure it is a good idea to guess something for an unbound type. I still think it would be better to use Any. Why do you think we should do something at all with unbound types?

@ilevkivskyi

ilevkivskyi Sep 1, 2017

Collaborator

I am still not sure it is a good idea to guess something for an unbound type. I still think it would be better to use Any. Why do you think we should do something at all with unbound types?

This comment has been minimized.

@dmoisset

dmoisset Sep 1, 2017

Contributor

Stubgen runs mostly at a syntactical level. Semantic analysis phase is not run within it. So essentially 99% of the type annotations it will find will still be unbound

@dmoisset

dmoisset Sep 1, 2017

Contributor

Stubgen runs mostly at a syntactical level. Semantic analysis phase is not run within it. So essentially 99% of the type annotations it will find will still be unbound

This comment has been minimized.

@ilevkivskyi

ilevkivskyi Sep 1, 2017

Collaborator

Stubgen runs mostly at a syntactical level. Semantic analysis phase is not run within it.

OK, is it easy to fix this? If not, then maybe add a # TODO: item here and/or open an issue?

@ilevkivskyi

ilevkivskyi Sep 1, 2017

Collaborator

Stubgen runs mostly at a syntactical level. Semantic analysis phase is not run within it.

OK, is it easy to fix this? If not, then maybe add a # TODO: item here and/or open an issue?

This comment has been minimized.

@dmoisset

dmoisset Sep 22, 2017

Contributor

I don't think it's easy to fix at all; it's probably a major change in the tool, the design decision of working syntactically is heavily embedded on it from day one.

@dmoisset

dmoisset Sep 22, 2017

Contributor

I don't think it's easy to fix at all; it's probably a major change in the tool, the design decision of working syntactically is heavily embedded on it from day one.

This comment has been minimized.

@ilevkivskyi

ilevkivskyi Sep 22, 2017

Collaborator

I would also have an issue for this. Maybe some day we will have time to improve this.

@ilevkivskyi

ilevkivskyi Sep 22, 2017

Collaborator

I would also have an issue for this. Maybe some day we will have time to improve this.

Show outdated Hide outdated mypy/stubgen.py
self._state = CLASS
def is_type_expression(self, expr: Expression, top_level: bool=True) -> bool:

This comment has been minimized.

@ilevkivskyi

ilevkivskyi Apr 27, 2017

Collaborator

I would like to see more tests for this, to be sure that complex (generic) aliases like Union[T, List[T]] are recognized, but things like type_map[int] are not.

@ilevkivskyi

ilevkivskyi Apr 27, 2017

Collaborator

I would like to see more tests for this, to be sure that complex (generic) aliases like Union[T, List[T]] are recognized, but things like type_map[int] are not.

This comment has been minimized.

@dmoisset

dmoisset Sep 1, 2017

Contributor

I can add a test for the former...
The latter will actually go through as an alias and I'm not sure how to prevent it with the kind of information I have in stubgen (I'm assuming that in most libraries that kind of things won't happen too much at a module top level).

@dmoisset

dmoisset Sep 1, 2017

Contributor

I can add a test for the former...
The latter will actually go through as an alias and I'm not sure how to prevent it with the kind of information I have in stubgen (I'm assuming that in most libraries that kind of things won't happen too much at a module top level).

Show outdated Hide outdated mypy/stubgen.py
Show outdated Hide outdated mypy/stubgen.py
Show outdated Hide outdated test-data/unit/stubgen.test
@@ -552,14 +602,148 @@ def f(a): ...
[case testInferOptionalOnlyFunc]
class A:
x = None
def __init__(self, a=None) -> None:

This comment has been minimized.

@ilevkivskyi

ilevkivskyi Apr 27, 2017

Collaborator

I would also keep an additional test for __init__.

@ilevkivskyi

ilevkivskyi Apr 27, 2017

Collaborator

I would also keep an additional test for __init__.

This comment has been minimized.

@dmoisset

dmoisset Sep 1, 2017

Contributor

Added

@dmoisset

dmoisset Sep 1, 2017

Contributor

Added

@ilevkivskyi

This comment has been minimized.

Show comment
Hide comment
@ilevkivskyi

ilevkivskyi Jul 15, 2017

Collaborator

@dmoisset Are you still working on this? I think this would be a useful addition (in particular in the context of helping maintainers of third-party libraries that want to support typing).

Collaborator

ilevkivskyi commented Jul 15, 2017

@dmoisset Are you still working on this? I think this would be a useful addition (in particular in the context of helping maintainers of third-party libraries that want to support typing).

@dmoisset

This comment has been minimized.

Show comment
Hide comment
@dmoisset

dmoisset Jul 22, 2017

Contributor

@ilevkivskyi I've been dealing with other stuff, I might take a look at this again during the rest of the month

Contributor

dmoisset commented Jul 22, 2017

@ilevkivskyi I've been dealing with other stuff, I might take a look at this again during the rest of the month

@JukkaL JukkaL removed their assignment Aug 15, 2017

@ilevkivskyi

This comment has been minimized.

Show comment
Hide comment
@ilevkivskyi

ilevkivskyi Aug 31, 2017

Collaborator

@dmoisset Sorry for pinging again, this is the oldest open PR now (and IMO it contains an important feature).

Collaborator

ilevkivskyi commented Aug 31, 2017

@dmoisset Sorry for pinging again, this is the oldest open PR now (and IMO it contains an important feature).

@ilevkivskyi ilevkivskyi self-assigned this Aug 31, 2017

@dmoisset

This comment has been minimized.

Show comment
Hide comment
@dmoisset

dmoisset Sep 1, 2017

Contributor

Thanks for the ping... I'm uploading a new version rebasing on a newer mypy and solving some of the items you mentioned (and replying to your comments about the items that need more discussion)

Contributor

dmoisset commented Sep 1, 2017

Thanks for the ping... I'm uploading a new version rebasing on a newer mypy and solving some of the items you mentioned (and replying to your comments about the items that need more discussion)

@ilevkivskyi

Thanks! This is almost ready, just few minor comments/questions.

Show outdated Hide outdated mypy/stubgen.py
Show outdated Hide outdated mypy/stubgen.py
Show outdated Hide outdated mypy/stubgen.py
Show outdated Hide outdated test-data/unit/stubgen.test
Show outdated Hide outdated test-data/unit/stubgen.test
noalias1: Any
noalias2: Any
noalias3: bool

This comment has been minimized.

@ilevkivskyi

ilevkivskyi Sep 1, 2017

Collaborator

Do you preserve @abstractmethod and other decorators? I think it is important to preserve @abstractmethod, @classmethod, @staticmethod, and @property.

@ilevkivskyi

ilevkivskyi Sep 1, 2017

Collaborator

Do you preserve @abstractmethod and other decorators? I think it is important to preserve @abstractmethod, @classmethod, @staticmethod, and @property.

This comment has been minimized.

@dmoisset

dmoisset Sep 22, 2017

Contributor

I didn't make any changes here wrt the original stubgen. I see some tests checking for property/staticmethod/classmethod being preserved, and other decorators removed (I think abstractmethod is not preserved). If you don't mind, I would add that as a separate issue to avoid scope creep here

@dmoisset

dmoisset Sep 22, 2017

Contributor

I didn't make any changes here wrt the original stubgen. I see some tests checking for property/staticmethod/classmethod being preserved, and other decorators removed (I think abstractmethod is not preserved). If you don't mind, I would add that as a separate issue to avoid scope creep here

This comment has been minimized.

@ilevkivskyi

ilevkivskyi Sep 22, 2017

Collaborator

OK, could you please open an issue for this?

@ilevkivskyi

ilevkivskyi Sep 22, 2017

Collaborator

OK, could you please open an issue for this?

@gvanrossum

This comment has been minimized.

Show comment
Hide comment
@gvanrossum

gvanrossum Sep 19, 2017

Member

Would love to see this land -- who's waiting for whom?

Member

gvanrossum commented Sep 19, 2017

Would love to see this land -- who's waiting for whom?

@ilevkivskyi

This comment has been minimized.

Show comment
Hide comment
@ilevkivskyi

ilevkivskyi Sep 19, 2017

Collaborator

I am waiting for @dmoisset to implement few last comments.

Collaborator

ilevkivskyi commented Sep 19, 2017

I am waiting for @dmoisset to implement few last comments.

dmoisset added a commit to dmoisset/peps that referenced this pull request Sep 22, 2017

@dmoisset

This comment has been minimized.

Show comment
Hide comment
@dmoisset

dmoisset Sep 22, 2017

Contributor

OK, I'm pushed a new update. There are a few issues that we've discussed in the comments that were actually issues in the previous version. Something that could be done if you agree is merge this and add these items as separate issues. What I have is:

  • There's no clear road to take to export names when __all__ is not defined #3169 (comment) (The difficulty here is deciding what to do, implementation is easy); I checked a bit what tests should be updated and there will be a lot of import foo as foo everywhere to add and I'm worried that in general it's not what's desired.
  • stubgen should do semantic analysis instead of dealing with unbounded types #3169 (comment) (this is a major change/rewrite)
  • stubgen considers as type aliases some assignments of values which may not be type aliases type_map[int] #3169 (comment) (this could be easier to mitigate also by doing more semantic analysis)
  • make sure abstractmethod is passed through #3169 (comment) (this should be a very small patch, but I didn't want to keep adding new stuff here)
Contributor

dmoisset commented Sep 22, 2017

OK, I'm pushed a new update. There are a few issues that we've discussed in the comments that were actually issues in the previous version. Something that could be done if you agree is merge this and add these items as separate issues. What I have is:

  • There's no clear road to take to export names when __all__ is not defined #3169 (comment) (The difficulty here is deciding what to do, implementation is easy); I checked a bit what tests should be updated and there will be a lot of import foo as foo everywhere to add and I'm worried that in general it's not what's desired.
  • stubgen should do semantic analysis instead of dealing with unbounded types #3169 (comment) (this is a major change/rewrite)
  • stubgen considers as type aliases some assignments of values which may not be type aliases type_map[int] #3169 (comment) (this could be easier to mitigate also by doing more semantic analysis)
  • make sure abstractmethod is passed through #3169 (comment) (this should be a very small patch, but I didn't want to keep adding new stuff here)
@dmoisset

This comment has been minimized.

Show comment
Hide comment
@dmoisset

dmoisset Sep 22, 2017

Contributor

(to clarify: I can add the issues, just wanted to sync on how we proceed)

Contributor

dmoisset commented Sep 22, 2017

(to clarify: I can add the issues, just wanted to sync on how we proceed)

@gvanrossum

This comment has been minimized.

Show comment
Hide comment
@gvanrossum

gvanrossum Sep 22, 2017

Member
Member

gvanrossum commented Sep 22, 2017

@dmoisset

This comment has been minimized.

Show comment
Hide comment
@dmoisset

dmoisset Sep 22, 2017

Contributor

Oh, the apologies are on me; code reviews have been very quick, and the delays have been on my side. My chances to work on this are limited so it can take me a long time to move forward after the reviews

Contributor

dmoisset commented Sep 22, 2017

Oh, the apologies are on me; code reviews have been very quick, and the delays have been on my side. My chances to work on this are limited so it can take me a long time to move forward after the reviews

@ilevkivskyi

This comment has been minimized.

Show comment
Hide comment
@ilevkivskyi

ilevkivskyi Sep 22, 2017

Collaborator

There's no clear road to take to export names when __all__ is not defined #3169 (comment) (The difficulty here is deciding what to do, implementation is easy)

The problem is that stubgen does not conform to PEP 484 both without and with your PR.
I think PEP 484 is clear here and we should just go with it, instead of reconsidering the as name rule again.

I checked a bit what tests should be updated and there will be a lot of import foo as foo everywhere to add and I'm worried that in general it's not what's desired.

These tests might be older than PEP 484 itself, so I am not surprised. We just need to update the tests, although I am fine to do this in a separate PR.

To summarize, I am fine with merging this PR as is to avoid unnecessary growth (it is already large). But, please open four separate issues for the problems you mentioned and mention this PR in every issue.

Collaborator

ilevkivskyi commented Sep 22, 2017

There's no clear road to take to export names when __all__ is not defined #3169 (comment) (The difficulty here is deciding what to do, implementation is easy)

The problem is that stubgen does not conform to PEP 484 both without and with your PR.
I think PEP 484 is clear here and we should just go with it, instead of reconsidering the as name rule again.

I checked a bit what tests should be updated and there will be a lot of import foo as foo everywhere to add and I'm worried that in general it's not what's desired.

These tests might be older than PEP 484 itself, so I am not surprised. We just need to update the tests, although I am fine to do this in a separate PR.

To summarize, I am fine with merging this PR as is to avoid unnecessary growth (it is already large). But, please open four separate issues for the problems you mentioned and mention this PR in every issue.

@ilevkivskyi ilevkivskyi merged commit 4fc4ae2 into python:master Sep 22, 2017

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
@ilevkivskyi

This comment has been minimized.

Show comment
Hide comment
@ilevkivskyi

ilevkivskyi Sep 22, 2017

Collaborator

@dmoisset
Thank you for working on this! This will make stubgen more usable, and I will be happy i fyou continue to work on the above mentioned issues.

Collaborator

ilevkivskyi commented Sep 22, 2017

@dmoisset
Thank you for working on this! This will make stubgen more usable, and I will be happy i fyou continue to work on the above mentioned issues.

@gvanrossum

This comment has been minimized.

Show comment
Hide comment
@gvanrossum

gvanrossum Sep 22, 2017

Member

Thanks so much Daniel for working on this! And Ivan for reviewing. Stubgen doesn't get enough developer love.

Member

gvanrossum commented Sep 22, 2017

Thanks so much Daniel for working on this! And Ivan for reviewing. Stubgen doesn't get enough developer love.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment