Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Final names and attributes #5522

Merged
merged 58 commits into from Sep 11, 2018
Merged

Final names and attributes #5522

merged 58 commits into from Sep 11, 2018

Conversation

@ilevkivskyi
Copy link
Collaborator

@ilevkivskyi ilevkivskyi commented Aug 22, 2018

Fixes #1214
Fixes python/typing#286
Fixes python/typing#242 (partially, other part is out of scope)

This is a working implementation of final access qualifier briefly discussed at PyCon typing meeting. Final names/attributes can be used to have more static guarantees about semantics of some code and can be used by other tools like mypyc for optimizations.

We can play with this implementation before starting to write an actual PEP.

The basic idea is simple: once declared as final, a name/attribute can't be re-assigned, overridden, or redefined in any other way. For example:

from typing import Final

NO: Final = 0
YES: Final = 255

class BaseEngine:
    RATE: Final[float] = 3000

YES = 1  # Error!

class Engine(BaseEngine):
    RATE = 9000  # Also an error!

For more use cases, examples, and specification, see the docs patch.

Here are some comments on decisions made:

  • What can be final? It is hard to say what semantic nodes are important, I started from just module and class constants, but quickly realized it is hard to draw the line without missing some use cases (in particular for mypyc). So I went ahead and implemented all of them, everything can be final: module constants, class-level and instance-level attributes, method, and also classes.
  • Two names or one name? I currently use two names Final for assignments and @final for decorators. My PEP8-formatted mind just can't accept @Final :-)
  • Should re-exported names keep they const-ness? I think yes, this is a very common pattern, so it looks like this is a sane default.
  • What to do with instance-level vs class-level attributes? The point here is that mypy has a common namespace for class attributes. I didn't want to complicate things (including the mental model), so I just decided that one can't have, e.g., a name that is constant on class but assignable on instances, etc. Such use cases are relatively rare, and we can implement this later if there will be high demand for this.

...deferred features:

  • I didn't implement any constant propagation in mypy yet. This can be done later on per use-case
    basis. For example:
    fields: Final = [('x', int), ('y', int)]
    NT = NamedTuple('NT', fields)
  • Should final classes be like sealed in Scala? I think probably no. On one hand it could be be a nice feature, on other hand it complicates the mental model and is less useful for things like mypyc.
  • I don't allow Final in function argument types. One argument is simplicity, another is I didn't see many bugs related to shadowing an argument in function bodies, finally people might have quite different expectations for this. If people will ask, this would be easy to implement.

...and implementation internals:

  • There are two additional safety nets that I don't mention in the docs: (a) there can be no TypeVars in the type of class-level constant, (b) instance-level constant can't be accessed on the class object.
  • I generate errors for re-definitions in all subclasses, not only in immediate children. I think this is what most people would want: turning something into a constant will flag most re-assignment points.
  • It is hard to always get a single error sometimes (I have two error messages, one for re-assignment, another for overriding) because of how mypy creates new Vars. But I think having two errors occasionally is fine.
  • We store the final_value for constants initialized with a simple literal, but we never use it. This exists only for tools like mypyc that may use it for optimizations.

The PR is open for questions, suggestions, and comments.

cc @ambv @rchen152 @vlasovskikh

@ilevkivskyi ilevkivskyi requested a review from msullivan Aug 22, 2018
@mvaled
Copy link

@mvaled mvaled commented Aug 22, 2018

@med-merchise you were asking for this, I think.

@Michael0x2a
Copy link
Collaborator

@Michael0x2a Michael0x2a commented Aug 22, 2018

This is super cool!

Just two comments -- first, it might be nice if there were some tests + docs about Final and control flow. For example, the following program typechecks without an error:

[case testFinalControlFlow]
from typing import Final

for val in [1, 2, 3]:
    x: Final = val

[builtins fixtures/list.pyi]

I found this a bit surprising since we are rebinding x several times/tools like mypyc can no longer assume that 'x' will have the same value within the loop. But then again, maybe this is ok? I think i'd be sort of hard to accidentally do something weird here, and any weirdness that does happen will be localized since x will be actually final after the loop is over...

Not sure, but either way, some clarification might be nice (unless you were planning on doing this in a different PR).

Second, I expect that this proposal will be rejected immediately for being too unpythonic + too messy to implemented, but I figure I might as well get it out of the way... How do we feel about adding a flag to mypy that makes all variables and attributes Final by default?

@ilevkivskyi
Copy link
Collaborator Author

@ilevkivskyi ilevkivskyi commented Aug 23, 2018

@Michael0x2a

first, it might be nice if there were some tests + docs about Final and control flow

I would actually prohibit Final in loops. Moreover, I would propose to also prohibit Final in if statements, unless the truthiness of condition is known statically (like sys.version > 3.6).

In addition, I just noticed that I forgot about in-place assignments, will fix this soon.

Second, I expect that this proposal will be rejected immediately for being too unpythonic

I am not sure I understand this comment. Rejected immediately by whom? There were no objections during PyCon typing meeting, moreover many people liked the idea.

too messy to implemented

Same story here, sorry :-) Do you want to say that the implementation in this PR is messy?

How do we feel about adding a flag to mypy that makes all variables and attributes Final by default?

I am not sure there is any existing Python program larger than 1k lines that will type-check with such flag. Plus we already have dozens of flags. So I am against this unless people will ask often for this.

@Michael0x2a
Copy link
Collaborator

@Michael0x2a Michael0x2a commented Aug 23, 2018

I am not sure I understand this comment. Rejected immediately by whom? There were no objections during PyCon typing meeting, moreover many people liked the idea.

Ah, I meant to say that I expected my proposal to add a flag to make final always-on by default would be rejected immediately by you.

(Which is what just happened -- but it was worth a shot 😄)

Copy link
Collaborator

@JukkaL JukkaL left a comment

Thanks for writing so many test cases! This is a solid set of test cases. I only came up with a few additional scenarios.

This is almost concludes my review. I'll do another light pass next week (I don't expect anything major).

self.check_final(s)
if (s.is_final_def and s.type and not has_no_typevars(s.type)
and self.scope.active_class() is not None):
self.fail("Constant declared in class body can't depend on type variables", s)

This comment has been minimized.

@JukkaL

JukkaL Sep 7, 2018
Collaborator

I think it would be better to not call final attributes constants. I.e. rename message to "Final attribute defined in ...".

mypy/checkmember.py Outdated Show resolved Hide resolved
mypy/messages.py Outdated Show resolved Hide resolved
mypy/messages.py Outdated Show resolved Hide resolved
mypy/semanal.py Outdated Show resolved Hide resolved
test-data/unit/check-final.test Show resolved Hide resolved
test-data/unit/check-final.test Outdated Show resolved Hide resolved
test-data/unit/check-final.test Show resolved Hide resolved
test-data/unit/check-final.test Show resolved Hide resolved
test-data/unit/check-final.test Outdated Show resolved Hide resolved
@ilevkivskyi
Copy link
Collaborator Author

@ilevkivskyi ilevkivskyi commented Sep 11, 2018

@JukkaL I implemented only some comments in the docs. I will kill it in the next commit, then you or any one else can finish it whenever.

@JukkaL
JukkaL approved these changes Sep 11, 2018
Copy link
Collaborator

@JukkaL JukkaL left a comment

Thanks for the updates! Looks good now, just one minor thing I noticed.

base: TypeInfo, base_node: Optional[Node]) -> bool:
"""Check if an assignment overrides a final attribute in a base class.
This only check situations where either a node in base class is not a variable

This comment has been minimized.

@JukkaL

JukkaL Sep 11, 2018
Collaborator

Grammar: check -> checks

@ilevkivskyi
Copy link
Collaborator Author

@ilevkivskyi ilevkivskyi commented Sep 11, 2018

OK, all tests passed, we are good to go. Thanks for review!

@ilevkivskyi ilevkivskyi merged commit 62e6f51 into python:master Sep 11, 2018
2 checks passed
2 checks passed
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
ilevkivskyi added a commit to python/typing that referenced this pull request Sep 13, 2018
This is a runtime counterpart of an experimental feature added to mypy in python/mypy#5522

This implementation just mimics the behaviour of `ClassVar` on all Python/`typing` versions, which is probably the most reasonable thing to do.
Michael0x2a added a commit to Michael0x2a/mypy that referenced this pull request May 7, 2019
This pull request is a follow-up to python#6763:
it renames some parameter names and variables that got missed up above.

One interesting side-note: it seems like 'Var' notes also contain a
`final_value` field that does almost the same thing as this
`last_known_value` field, but is ultimately unrelated: it was introduced
back in python#5522. I decided to leave
this field alone.

(I discovered this while updating mypyc and discovered (to my surprise)
that nothing broke.)
ilevkivskyi added a commit that referenced this pull request May 8, 2019
This pull request is a follow-up to #6763:
it renames some parameter names and variables that got missed up above.

One interesting side-note: it seems like 'Var' notes also contain a
`final_value` field that does almost the same thing as this
`last_known_value` field, but is ultimately unrelated: it was introduced
back in #5522. I decided to leave
this field alone.

(I discovered this while updating mypyc and discovered (to my surprise)
that nothing broke.)
@yxliang01
Copy link

@yxliang01 yxliang01 commented Aug 20, 2019

From python official documentation for Python3.7, I don't see Final being mentioned anywhere. While executing python file with from typing import Final under Python3.6 , it raised ImportError: cannot import name 'Final'. As a result, this is not really usable? Because this is only valid when we lint using "mypy" and we even need to remove all occurrences of Final before execution.

@ilevkivskyi ilevkivskyi deleted the ilevkivskyi:final branch Aug 20, 2019
@ilevkivskyi
Copy link
Collaborator Author

@ilevkivskyi ilevkivskyi commented Aug 20, 2019

@yxliang01 It will be available in typing starting from Python 3.8, on older versions you should use typing_extensions. There is a actually a big blue box saying this in the docs. Next time please read them first, if you have troubles understanding how to use mypy.

@yxliang01
Copy link

@yxliang01 yxliang01 commented Aug 20, 2019

Thanks for clarification!
p.s. I don't think I have difficulty on using mypy(besides with this Final thing) and I was directly following the repos and found this PR, that's why I missed that doc :(

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
7 participants