Last strict optional fixes #4070

ilevkivskyi · 2017-10-08T00:55:34Z

Fixes #1955

This PR will make mypy completely --strict-optional clean. In addition to signature fixes and other minor fixes (mostly in semanal.py) I introduce a change in mypy as a typechecker, I have noticed a pattern that appears several times, but was not understood by mypy:

x: Optional[int]
if x in [1, 2, 3]:
    reveal_type(x)  # now this will be 'int'

I allow this only for built-in containers such as list, dict, set, frozenset, and tuple, since other types can change the semantics of __contains__. (Also this way it is a small change, a general treatment of Container subclasses will require code churn to get named_type).

UPDATE: The feature part now has a separate PR #4072.

…nal-fixes Conflicts: mypy/checker.py mypy/checkexpr.py

gvanrossum · 2017-10-08T01:35:28Z

How hard would it be to separate the new feature from the rest of the PR? (And describe it in an issue while you're at it?)

JelleZijlstra · 2017-10-08T01:53:17Z

mypy/checker.py

+    # This is only OK for built-in containers, where we know the behavior of __contains__.
+    if isinstance(tp, Instance):
+        if tp.type.fullname() in ['builtins.list', 'builtins.tuple', 'builtins.dict',
+                                  'builtins.set', 'builtins.frozenfet']:


frozenset, not frozenfet

JelleZijlstra · 2017-10-08T01:56:27Z

mypy/semanal.py

-                                                           [defn.type.ret_type]))
+                                                           [defn.type.ret_type]) or
+                        # We are running tests
+                        self.named_type('typing_full.Awaitable', [defn.type.ret_type]))


This should be indented more (and maybe rearranged; I think it's too much for one statement).

ilevkivskyi · 2017-10-08T11:23:27Z

@gvanrossum

How hard would it be to separate the new feature from the rest of the PR? (And describe it in an issue while you're at it?)

OK, I opened #4071 and separated the feature plus added tests in #4072. I would like to keep the code here, so that we can avoid:

if fullname in ('enum.Enum', 'enum.IntEnum'):
    assert fullname is not None

etc. The PRs can be merged in arbitrary order, however it makes sense to merge #4072 first.

…nal-fixes Conflicts: mypy/semanal.py mypy_self_check.ini

ilevkivskyi · 2017-10-13T15:05:52Z

@JukkaL While this is fresh could you please review also this PR (to avoid more conflicts)?

JukkaL · 2017-10-13T15:33:03Z

Sure, I'll give it a go.

gvanrossum · 2017-10-13T20:02:32Z

Dang, there's already another conflict. :-(

ilevkivskyi · 2017-10-13T20:03:38Z

Should be fixed now.

gvanrossum · 2017-10-13T20:20:23Z

Thanks, I'm going to leave the review to Jukka (who's busy moving this weekend).

JukkaL

Thanks, looks good! This migration has been a big undertaking and it's great to see the final strict optional exceptions removed. I left just some minor comments.

JukkaL · 2017-10-13T15:46:29Z

mypy/semanal.py

@@ -393,6 +396,7 @@ def visit_func_def(self, defn: FuncDef) -> None:
                # Top-level function
                if not defn.is_decorated and not defn.is_overload:
                    symbol = self.globals.get(defn.name())
+                    assert symbol, "Global function not found in globals"


We could instead just use self.globals[defn.name()] above?

JukkaL · 2017-10-13T15:48:57Z

mypy/semanal.py

+                    ret_type = self.named_type_or_none('typing.Awaitable', [defn.type.ret_type])
+                    if ret_type is None:
+                        # We are running tests.
+                        ret_type = self.named_type('typing_full.Awaitable', [defn.type.ret_type])


What is typing_full? Could we just blow up if typing.Awaitable is not available?

Without this line many tests fail (see the comment just above). I think this is because those tests use fixtures/typing-full.py. A better fix would be to always name it typing in tests, but I think this should be a separate PR.

So the issue is that typing-full.py is imported as module named typing_full in tests? If this is true, this should be fixed (in another PR).

Hm, my initial analysis is wrong, the problem is how named_type works. I will make a PR now.

OK, opened #4158

JukkaL · 2017-10-13T15:51:11Z

mypy/semanal.py

            self.errors.pop_function()

    def prepare_method_signature(self, func: FuncDef) -> None:
        """Check basic signature validity and tweak annotation of self/cls argument."""
        # Only non-static methods are special.
+        assert self.type is not None, "This method should be only called inside a class"


You could remove this assert by adding a TypeInfo argument and passing it from the caller.

JukkaL · 2017-10-13T15:53:15Z

mypy/semanal.py

@@ -534,7 +541,7 @@ def visit_overloaded_func_def(self, defn: OverloadedFuncDef) -> None:
                assert defn.impl is defn.items[-1]
                defn.items = defn.items[:-1]
            elif not self.is_stub_file and not non_overload_indexes:
-                if not (self.is_class_scope() and self.type.is_protocol):
+                if not (self.type and not self.is_func_scope() and self.type.is_protocol):


Random observation: Here it would be useful is is_class_scope() could return the binding self.type is not None when it returns true (there is a feature proposal about this, I think).

Yes, I also noticed it would be very useful to somehow "remember" the bindings. For simple cases I was thinking about something like inlining methods (very approximately):

@inline def meth(self) -> bool: return self.x is not None and self.y is not None if self.meth(): # use original expression returned by 'meth' here ...

JukkaL · 2017-10-13T15:55:03Z

mypy/semanal.py

@@ -690,8 +696,10 @@ def analyze_class_body(self, defn: ClassDef) -> Iterator[bool]:
                    if prohibited in named_tuple_info.names:
                        if nt_names.get(prohibited) is named_tuple_info.names[prohibited]:
                            continue
+                        ctx = named_tuple_info.names[prohibited].node
+                        assert ctx is not None


What if the definition is a bound type variable where node is None, I think? In a separate PR, it would be nice to make the node attribute non-optional (this is something we've discussed before).

What if the definition is a bound type variable where node is None, I think?

Currently generic named tuples are not supported, so I thought this should never happen.
I would keep this assert and then if it ever fails it will be clear what happens there.

JukkaL · 2017-10-13T15:59:13Z

mypy/semanal.py

            if isinstance(defn.metaclass, NameExpr):
                metaclass_name = defn.metaclass.name
            elif isinstance(defn.metaclass, MemberExpr):
                metaclass_name = get_member_expr_fullname(defn.metaclass)
-            else:
+            if metaclass_name is None:


Ah does this fix a bug, in case get_member_expr_fullname returns None? If yes, can you add a test case for the original issue?

Indeed, this fixes a crash (although it was not my intention, I just fixed the optional type here). I will add a test.

JukkaL · 2017-10-20T10:33:12Z

mypy/semanal.py

@@ -2576,6 +2595,7 @@ def parse_typeddict_args(self, call: CallExpr,
            for t in types:
                if has_any_from_unimported_type(t):
                    self.msg.unimported_type_becomes_any("Type of a TypedDict key", t, dictexpr)
+        assert total is not None


This seems redundant, since we return if total is None above.

Not from mypy's point of view. We return if total is None and len(args) == 3. I think this could be fixed by #2008

ilevkivskyi · 2017-10-20T22:55:03Z

@JukkaL Thanks for review! I pushed fixes in a new commit.

JukkaL

Looks good now! Thanks for finishing the strict-optional migration!

This makes mypy completely --strict-optional clean. Fixes #1955.

In #4070, `CallableType.__init__` was changed to accept `arg_names: Sequence[str | None]` so we could pass e.g. `list[str]` to it. We're making a similar change to `CallableType.copy_modified`.

ilevkivskyi added 13 commits September 14, 2017 13:20

Clean-up most of checker; started checkexpr

178ad41

More clean-up

9d77fbc

Finish clean-up of checker, checkexpr and myunit

1f73106

Address CR

218253a

Fix some more strict optional errors

233c298

Merge remote-tracking branch 'upstream/master' into last-strict-optio…

cd6613d

…nal-fixes Conflicts: mypy/checker.py mypy/checkexpr.py

Deal with self.type

583c00e

Deal with lookup functions

6ecd80b

Fix most signatures and other fixes

b105456

Allow optional removal after container 'in' checks

363407c

More fixes

eaa6edb

More fixes

0302811

Generalize/simplify container check

6387359

JelleZijlstra reviewed Oct 8, 2017

View reviewed changes

Address CR

424334a

ilevkivskyi mentioned this pull request Oct 8, 2017

Narrow types after 'in' operator #4072

Merged

ilevkivskyi added 3 commits October 8, 2017 13:29

Sync with python#4072

80c1b38

Sync with python#4072 for realz

2a96e50

Merge branch 'master' into last-strict-optional-fixes

6f8cd68

This was referenced Oct 9, 2017

Release 0.540 planning #4084

Closed

Refactor: Move semantic analyzer pass 3 to a new module #4099

Merged

ilevkivskyi added 3 commits October 13, 2017 16:27

Merge remote-tracking branch 'upstream/master' into last-strict-optio…

46e8c2d

…nal-fixes Conflicts: mypy/semanal.py mypy_self_check.ini

Fix merge and sync with python#4072

a877957

Fix typeshed commit

9969ecb

Merge branch 'master' into last-strict-optional-fixes

aabbefa

JukkaL reviewed Oct 20, 2017

View reviewed changes

Address CR

2549386

JukkaL approved these changes Oct 24, 2017

View reviewed changes

JukkaL merged commit 936ceac into python:master Oct 24, 2017

ilevkivskyi deleted the last-strict-optional-fixes branch October 25, 2017 08:21

ilevkivskyi mentioned this pull request Oct 25, 2017

Remove the typing_full hack #4158

Merged

JukkaL pushed a commit that referenced this pull request Oct 31, 2017

Last strict optional fixes (#4070)

0c12522

This makes mypy completely --strict-optional clean. Fixes #1955.

ikonst mentioned this pull request Mar 6, 2023

CallableType.copy_modified: accept sequence for arg_names #14840

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Last strict optional fixes #4070

Last strict optional fixes #4070

ilevkivskyi commented Oct 8, 2017 •

edited

Loading

gvanrossum commented Oct 8, 2017 via email

JelleZijlstra Oct 8, 2017

JelleZijlstra Oct 8, 2017

ilevkivskyi commented Oct 8, 2017

ilevkivskyi commented Oct 13, 2017

JukkaL commented Oct 13, 2017

gvanrossum commented Oct 13, 2017

ilevkivskyi commented Oct 13, 2017

gvanrossum commented Oct 13, 2017 via email

JukkaL left a comment

JukkaL Oct 13, 2017

JukkaL Oct 13, 2017

ilevkivskyi Oct 20, 2017

JukkaL Oct 24, 2017

ilevkivskyi Oct 25, 2017

ilevkivskyi Oct 25, 2017

JukkaL Oct 13, 2017

JukkaL Oct 13, 2017

ilevkivskyi Oct 20, 2017 •

edited

Loading

JukkaL Oct 13, 2017

ilevkivskyi Oct 20, 2017

JukkaL Oct 13, 2017

ilevkivskyi Oct 20, 2017

JukkaL Oct 20, 2017

ilevkivskyi Oct 20, 2017

ilevkivskyi commented Oct 20, 2017

JukkaL left a comment

Last strict optional fixes #4070

Last strict optional fixes #4070

Conversation

ilevkivskyi commented Oct 8, 2017 • edited Loading

gvanrossum commented Oct 8, 2017 via email

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ilevkivskyi commented Oct 8, 2017

ilevkivskyi commented Oct 13, 2017

JukkaL commented Oct 13, 2017

gvanrossum commented Oct 13, 2017

ilevkivskyi commented Oct 13, 2017

gvanrossum commented Oct 13, 2017 via email

JukkaL left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ilevkivskyi Oct 20, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ilevkivskyi commented Oct 20, 2017

JukkaL left a comment

Choose a reason for hiding this comment

ilevkivskyi commented Oct 8, 2017 •

edited

Loading

ilevkivskyi Oct 20, 2017 •

edited

Loading