New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Detect and support module aliasing via assignment. #3435
Changes from 15 commits
ad646b9
c498e2a
fdb3af1
14d7d0f
ee8bd80
7efbdb7
fd3eb84
4006d0a
7aeb65f
6dc0757
9aa8169
6e7cd24
d3d8f9f
1cbe94c
325ae94
2326425
9592ce9
f438f9c
e1ce5b8
c51a916
4d40207
a8a4f44
6584b0b
b292673
f62d58e
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -1520,6 +1520,7 @@ def visit_assignment_stmt(self, s: AssignmentStmt) -> None: | |
self.process_namedtuple_definition(s) | ||
self.process_typeddict_definition(s) | ||
self.process_enum_call(s) | ||
self.process_module_assignment(s) | ||
|
||
if (len(s.lvalues) == 1 and isinstance(s.lvalues[0], NameExpr) and | ||
s.lvalues[0].name == '__all__' and s.lvalues[0].kind == GDEF and | ||
|
@@ -2356,6 +2357,64 @@ def is_classvar(self, typ: Type) -> bool: | |
def fail_invalid_classvar(self, context: Context) -> None: | ||
self.fail('ClassVar can only be used for assignments in class body', context) | ||
|
||
def process_module_assignment(self, s: AssignmentStmt) -> None: | ||
"""Check if s assigns a module an alias name; if so, update symbol table.""" | ||
self._process_module_assignment(s.lvalues, s.rvalue, s) | ||
|
||
def _process_module_assignment( | ||
self, | ||
lvals: List[Expression], | ||
rval: Expression, | ||
ctx: AssignmentStmt, | ||
) -> None: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Typically this style is not used in mypy. You can just wrap argument list like this: def process_module_assignment(self, lvals: List[Expression], rval: Expression,
ctx: AssignmentStmt) -> None: |
||
"""Propagate module references across assignments. | ||
|
||
Recursively handles the simple form of iterable unpacking; doesn't | ||
handle advanced unpacking with *rest, dictionary unpacking, etc. | ||
|
||
In an expression like x = y = z, z is the rval and lvals will be [x, | ||
y]. | ||
|
||
""" | ||
if all(isinstance(v, (TupleExpr, ListExpr)) for v in lvals + [rval]): | ||
# rval and all lvals are either list or tuple, so we are dealing | ||
# with unpacking assignment like `x, y = a, b`. Mypy didn't | ||
# understand our all(isinstance(...)), so cast them as | ||
# Union[TupleExpr, ListExpr] so mypy knows it is safe to access | ||
# their .items attribute. | ||
seq_lvals = cast(List[Union[TupleExpr, ListExpr]], lvals) | ||
seq_rval = cast(Union[TupleExpr, ListExpr], rval) | ||
# given an assignment like: | ||
# (x, y) = (m, n) = (a, b) | ||
# we now have: | ||
# seq_lvals = [(x, y), (m, n)] | ||
# seq_rval = (a, b) | ||
# We now zip this into: | ||
# elementwise_assignments = [(a, x, m), (b, y, n)] | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maybe put |
||
# where each elementwise assignment includes one element of rval and the | ||
# corresponding element of each lval. Basically we unpack | ||
# (x, y) = (m, n) = (a, b) | ||
# into elementwise assignments | ||
# x = m = a | ||
# y = n = b | ||
# and then we recursively call this method for each of those assignments. | ||
# If the rval and all lvals are not all of the same length, zip will just ignore | ||
# extra elements, so no error will be raised here; mypy will later complain | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is somehow not convincing. Could you please add a test that import m
x, y = m, m, m is flagged as an error? (Plus also some length mismatches in nested scenarios.) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yesterday I added a test for the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. ... and also for length mismatches in nested scenarios. |
||
# about the length mismatch in type-checking. | ||
elementwise_assignments = zip(seq_rval.items, *[v.items for v in seq_lvals]) | ||
for rv, *lvs in elementwise_assignments: | ||
self._process_module_assignment(lvs, rv, ctx) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think this looks a bit unclear and probably unnecessary. We need to support only There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can you be more specific what you find unclear about this code so I can improve the clarity? When I rewrite this code to not support nesting, it doesn't make it any simpler or clearer. The recursive approach is still the simplest even for a single layer of iterable (otherwise you have to do something else weird to support both iterable and single rval), and once you have recursion, not supporting nesting is more complex than supporting it. I agree that nested unpacked module assignment is likely very rare, but it seems strange to me to reduce compatibility with true Python semantics in order to achieve (AFAICS) a negligible or nonexistent gain in clarity/simplicity. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I have to agree that with all the comprehensions, casts, repetitions of (TupleExpr, ListExpr) and cleverness with zip and *x, I have no confidence that I understand what this code does. And then I'm not even touching on recursion. So the code appears unmaintainable, and I think it needs to be improved (maybe some comments would help). There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ok! Pushed an update that breaks down the code a bit and adds comments to clarify the intent. Let me know if that helps. |
||
elif isinstance(rval, NameExpr): | ||
rnode = self.lookup(rval.name, ctx) | ||
if rnode and rnode.kind == MODULE_REF: | ||
for lval in lvals: | ||
if not isinstance(lval, NameExpr): | ||
continue | ||
lnode = self.lookup(lval.name, ctx) | ||
if lnode: | ||
lnode.kind = MODULE_REF | ||
lnode.node = rnode.node | ||
|
||
def process_enum_call(self, s: AssignmentStmt) -> None: | ||
"""Check if s defines an Enum; if yes, store the definition in symbol table.""" | ||
if len(s.lvalues) != 1 or not isinstance(s.lvalues[0], NameExpr): | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -23,7 +23,7 @@ def safe_rmdir(dirname: str) -> None: | |
|
||
class GenericTest(unittest.TestCase): | ||
# The path module to be tested | ||
pathmodule = genericpath # type: Any | ||
pathmodule = genericpath | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think you should keep the import m
import types
class A: ...
class B(A): ...
Alias = A # creates a type alias
Var: Type[A] = A # creates a normal variable
mod = m # creates a module alias
mod.a # OK
mod_var: types.ModuleType = m # creates a normal variable
mod_var.a # Fails, ModuleType has no attribute a And don't forget to add a test for module assignment with an explicit type (both There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ok, so taking several things separately:
This code type-checks fine, and reveals a type of
I would definitely expect the latter code to succeed and reveal type of I've pushed the behavior that makes sense to me for now; will be happy to update once I understand why. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Hmm, after further thought I think I understand your point: "type alias" is to "var of type Type" as "module alias" is to "var of type types.ModuleType." I'm still not sure that this provides the most intuitive or useful behavior, given that type aliases and module references are used in different ways. I just tested and I can still pass a real module reference to a function that takes a parameter annotated as Anyway, changed my mind and pushing all changes as you suggest. Still would like to understand the reasoning, though. |
||
common_attributes = ['commonprefix', 'getsize', 'getatime', 'getctime', | ||
'getmtime', 'exists', 'isdir', 'isfile'] | ||
attributes = [] # type: List[str] | ||
|
@@ -143,7 +143,7 @@ def test_exists(self) -> None: | |
f.close() | ||
self.assertIs(self.pathmodule.exists(support.TESTFN), True) | ||
if not self.pathmodule == genericpath: | ||
self.assertIs(self.pathmodule.lexists(support.TESTFN), | ||
self.assertIs(self.pathmodule.lexists(support.TESTFN), # type: ignore | ||
True) | ||
finally: | ||
if not f.closed: | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -1439,3 +1439,82 @@ class C: | |
a = 'foo' | ||
|
||
[builtins fixtures/module.pyi] | ||
|
||
[case testModuleAssignment] | ||
import m | ||
m2 = m | ||
reveal_type(m2.a) # E: Revealed type is 'builtins.str' | ||
m2.b # E: Module has no attribute "b" | ||
m2.c = 'bar' # E: Module has no attribute "c" | ||
|
||
[file m.py] | ||
a = 'foo' | ||
|
||
[builtins fixtures/module.pyi] | ||
|
||
[case testClassModuleAssignment] | ||
import m | ||
|
||
class C: | ||
x = m | ||
def foo(self) -> None: | ||
reveal_type(self.x.a) # E: Revealed type is 'builtins.str' | ||
|
||
[file m.py] | ||
a = 'foo' | ||
|
||
[builtins fixtures/module.pyi] | ||
|
||
[case testLocalModuleAssignment] | ||
import m | ||
|
||
def foo() -> None: | ||
x = m | ||
reveal_type(x.a) # E: Revealed type is 'builtins.str' | ||
|
||
class C: | ||
def foo(self) -> None: | ||
x = m | ||
reveal_type(x.a) # E: Revealed type is 'builtins.str' | ||
|
||
[file m.py] | ||
a = 'foo' | ||
|
||
[builtins fixtures/module.pyi] | ||
|
||
[case testChainedModuleAssignment] | ||
import m | ||
m3 = m2 = m | ||
m4 = m3 | ||
m5 = m4 | ||
reveal_type(m2.a) # E: Revealed type is 'builtins.str' | ||
reveal_type(m3.a) # E: Revealed type is 'builtins.str' | ||
reveal_type(m4.a) # E: Revealed type is 'builtins.str' | ||
reveal_type(m5.a) # E: Revealed type is 'builtins.str' | ||
|
||
[file m.py] | ||
a = 'foo' | ||
|
||
[builtins fixtures/module.pyi] | ||
|
||
[case testMultiModuleAssignment] | ||
import m, n | ||
m2, n2, (m3, n3) = m, n, [m, n] | ||
reveal_type(m2.a) # E: Revealed type is 'builtins.str' | ||
reveal_type(n2.b) # E: Revealed type is 'builtins.str' | ||
reveal_type(m3.a) # E: Revealed type is 'builtins.str' | ||
reveal_type(n3.b) # E: Revealed type is 'builtins.str' | ||
|
||
x, y = m # E: 'types.ModuleType' object is not iterable | ||
x, y, z = m, n # E: Need more than 2 values to unpack (3 expected) | ||
x, y = m, m, m # E: Too many values to unpack (2 expected, 3 provided) | ||
x, (y, z) = m, n # E: 'types.ModuleType' object is not iterable | ||
x, (y, z) = m, (n, n, n) # E: Too many values to unpack (2 expected, 3 provided) | ||
|
||
[file m.py] | ||
a = 'foo' | ||
|
||
[file n.py] | ||
b = 'bar' | ||
|
||
[builtins fixtures/module.pyi] | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I would like to see few more tests. For example, a test that shows this failing: import m
x, y = m # must be an error here (Module object is not iterable or similar) and in general more failures, like failure on attempted access/assignment to a non-existing module attribute. Also probably some chained assignments: import m
x = m
y = x
reveal_type(y.a) # 'builtins.str' There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 👍 Very good point, will add these tests. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do you need a function that just unconditionally calls another function? Maybe one is enough?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, good point. This was to present a cleaner API (pass in only the
AssignmentStmt
), but given there's only one call site and will never be more, that's not very useful. Consolidated.