Strict Optional checking

[ddfisher]

(copied from internal discussion)

Proposal

With one exception, make MyPy enforce optional semantics as specified in PEP 484:

By default, None is an invalid value for any type, unless a default value of None has been provided in the function definition.
…
As a shorthand for Union[T1, None] you can write Optional[T1].

The one exception will be in the case of instance variables defined in the class body. These variables may be assigned None while having a non-Optional type, provided that they are initialized to a non-None value by the end of __init__. They may be initialized in __init__ itself, or in any method on self that __init__ calls, or in __new__. Mypy will not check these variables for use before assignment to a non-None value in __init__ or called methods, though it may make sense to add that at some point in the future.
The purpose of this exception is to allow variables to be typed in a common place rather than scattered around __init__.

Here’s how this would look in particular:

Make None a real type

It supports a limited set of operations. Optional[x] is the same as Union[x, None], and this is generally different from just x.

    def f(x: Optional[str]) -> None:
        x.startswith('x')   # error, None has no startswith

    def g(x: str) -> None: ...

    g(None)  # error

Infer Optional[...] from None default value

    def f(x: str = None) -> None:
        # type of x is Optional[str]
        ...

    f('')   # ok
    f(None) # ok

If [not] value check

    def f(x: str = None) -> None:
        if x:
            # type of x should be str here
            x.startswith('x')  # should be fine
        else:
            # type of x should be Optional[str] here
            x.startswith('x')  # error

If value is [not] None check

    def f(x: str = None) -> None:
        if x is not None:
            # type of x should be str here
            x.startswith('x')  # should be fine
        else:
            # type of x should be None here
            x.startswith('x')  # error

assert x

    def f(x: str = None) -> None:
        assert x
        x.startswith('x')  # should be fine

Similar for assert [not] x

or/and

    def f(x: str = None, y: str = None) -> None:
        x and x.startswith('x')          # should be fine (and easy)
        x is None or x.startswith('x')   # should be fine
        x and y and x.startswith('x') and y.startswith('x')   # fine

    def g(x: str = None) -> str:
        return x or "default"

Generators/comprehensions with condition check

    a = ... # type: List[Optional[int]]
    b = [x for x in b if x]
    # type of b should be List[int]

Overloading with None

The new overload syntax (I can’t remember whether it’s in PEP 484 already) would allow overloading based on None values:

    @overload
    def f(x: None) -> None: ...
    @overload
    def f(x: str) -> int: ...

    def f(x):
        if x:
            return int(x)
        else:
            return None

    f('')    # type is str
    f(None)  # type is None

Class Variable that Gets Initialized to None

    class A:
        x = None  # type: str
        def __init__(self) -> None:
            x = "value"

    class B:
        x = None  # type: str
        def __init__(self) -> None:
            self.setup()

        def setup(self) -> None:
            x = "value"

Attribute that Gets Initialized to None

    class A:
        x = None  # type: str
        def __init__(self) -> None:
            self.x = None  # type: ignore

        def load(self, f: IO[str]) -> None:
            self.x = f.read()

        def process(self) -> None:
            print(x + 'x')  # should be fine?

    class B:
        def __init__(self) -> None:
            self.x = None  # type: Optional[str]

        def load(self, f: IO[str]) -> None:
            self.x = f.read()

        def process(self) -> None:
            assert self.x    # needed, otherwise next line is an error
            print(x + 'x')   # fine

Future/Out of Scope Work for Initial Version

Functions without Explicit Return with Non-Optional Return Type

We’ll need to do some control flow analysis to handle cases like this, so we shouldn't implement this initially.

    def f(x: int) -> int:
      if x > 0:
        return 5
      else:
        pass  # uh oh, no return value here

Development Plan

Will probably need to start with #1278 as a first step (but not something to be merged separately). Will be behind a feature flag initially and turned on by default later.

[porglezomp]

We’ll need to do some control flow analysis to handle cases like this, so we shouldn't implement this initially.

Wouldn't the typing on member initialization also rely on similar control flow analysis? (e.g. in "Attribute that Gets Initialized to None," etc.)

[gvanrossum]

Can you open new issues for the follow-up tasks? At least:

• make mypy itself optional-clean
• make our internal regresssion tests optional-clean
• make mypy's tests optional-clean
• [optionally] make this the default behavior (leaving "non-strict optional" as a flag)
• make this the only behavior (removing the latter flag again)

[revmischa]

if I have an Optional[str] and then check to see if it's defined, I can't pass it to a function that expects str. How should this work?

[ethanhs]

@revmischa you probably want to read https://mypy.readthedocs.io/en/latest/kinds_of_types.html#experimental-strict-optional-type-and-none-checking

[revmischa]

Thanks! There was some implicit stuff going on that tripped me up.