mypy/typing false positives

[ojii]

Using attrs 6e7b9f2 and mypy 0.720:

from typing import List, Dict, Any, Union, Optional
import attr


@attr.s
class Foo:
    # OK
    str_list: List[str] = attr.ib(
        validator=attr.validators.deep_iterable(
            attr.validators.instance_of(str), attr.validators.instance_of(list)
        )
    )
    # Incompatible types in assignment (expression has type "Optional[List[_T]]", variable has type "Optional[List[str]]")
    maybe_str_list: Optional[List[str]] = attr.ib(
        validator=attr.validators.optional(
            attr.validators.deep_iterable(
                attr.validators.instance_of(str), attr.validators.instance_of(list)
            )
        )
    )
    # OK
    str_dict: Dict[str, Any] = attr.ib(
        validator=attr.validators.deep_mapping(
            attr.validators.instance_of(str),
            attr.validators.instance_of(object),
            attr.validators.instance_of(dict),
        )
    )
    # OK
    maybe_str_dict: Optional[Dict[str, Any]] = attr.ib(
        validator=attr.validators.optional(
            attr.validators.deep_mapping(
                attr.validators.instance_of(str),
                attr.validators.instance_of(object),
                attr.validators.instance_of(dict),
            )
        )
    )
    # Argument 1 to "instance_of" has incompatible type "Tuple[Type[str], Type[int]]"; expected "Union[Tuple[Type[<nothing>], ...], Type[<nothing>]]"
    str_int: Union[str, int] = attr.ib(
        validator=attr.validators.instance_of((str, int))
    )
    # (the error is shown twice for some reason)
    # Argument 1 to "instance_of" has incompatible type "Tuple[Type[str], Type[int]]"; expected "Union[Tuple[Type[<nothing>], ...], Type[<nothing>]]"
    # Argument 1 to "instance_of" has incompatible type "Tuple[Type[str], Type[int]]"; expected "Union[Tuple[Type[None], ...], Type[None]]"
    maybe_str_int: Optional[Union[str, int]] = attr.ib(
        validator=attr.validators.optional(attr.validators.instance_of((str, int)))
    )

deep_iterable seems to have an issue when combined with optional, instance_of doesn't like tuples of types.

[hynek]

Paging @mmaslowskicc @euresti @ethanhs 🤞

[euresti]

instance_of can be fixed with @overloads like this:

_T = TypeVar("_T")
_T2 = TypeVar("_T2")
_T3 = TypeVar("_T3")

@overload
def instance_of(type: Type[_T]) -> attr._ValidatorType[_T]:
    ...

@overload
def instance_of(type: Tuple[Type[_T]]) -> attr._ValidatorType[_T]:
    ...

@overload
def instance_of(type: Tuple[Type[_T], Type[_T2]]) -> attr._ValidatorType[Union[_T, _T2]]:
    ...

@overload
def instance_of(type: Tuple[Type[_T], Type[_T2], Type[_T3]]) -> attr._ValidatorType[Union[_T, _T2, _T3]]:
    ...

# Ok just fall back to Any if more than 3 values.
@overload
def instance_of(type: Tuple[type, ...]) -> attr._ValidatorType[Any]:
    ...

But how many do we add before giving up?

[hynek]

Three sounds like a good number?

[euresti]

I see the issue with deep_iterable but I don't know how to fix it.

deep_iterable is typed like this:

def deep_iterable(
    member_validator: _ValidatorType[_T],
    iterable_validator: Optional[_ValidatorType[_I]] = ...,
) -> _ValidatorType[_I]: ...

list is technically a Type[List[_T]]
which makes instance_of(list) a _ValidatorType[List[_T]]

Now when you add optional you end up with an Optional[List[_T]] which for some reason mypy won't coerce to an Optional[List[str]] even though it seems to do it just fine to do List[_T] to List[str]. I don't know why Optional is working against you there.

I see some paths forward:
A. Leave things as is. Users can always use typing.cast to work around the issue.
B. Have deep_iterable return a _ValidatorType[Any]. We lose some type checking but this is seriously weird code.
C. Spend several hours/days figuring out why the Optional[List[_T]] doesn't coerce to Optional[List[str]] issue in the mypy code and fix it.
D Spend several hours adding some code to the plugin to make deep_iterable return List[str] for this particular flow.

I don't have too much time so I cannot work on C or D.
A is free. B is just a simple stubs change.

[hynek]

Maybe A + report a bug to mypy?

[ojii]

A. Leave things as is. Users can always use typing.cast to work around the issue.

how can typing.cast be used in the case of a Optional[List[str]] in attrs to make mypy happy?

Answering my own question:

    maybe_str_list: Optional[List[str]] = attr.ib(
        validator=cast(
            attr._ValidatorType[Optional[List[str]]],
            attr.validators.optional(
                attr.validators.deep_iterable(
                    attr.validators.instance_of(str), attr.validators.instance_of(list)
                )
            ),
        )
    )```

quite the sight but it pacifies mypy

[euresti]

Yes, but you'll want to put attr._ValidatorType[Optional[List[str]]] in quotes because that's only defined in the stub and not at runtime.

Can the cast go on the attr.ib or does that break the plugin?

[hynek]

Can you confirm instance_of is fixed, Jonas?

[hynek]

@euresti can I bother you to open a bug on mypy regarding the other issue please? 🐶 I believe we can close this bug once that's done.

[hawkowl]

ran into this, and attrs master fixes the instance_of being given a tuple.

[euresti]

Hmm. I now get no issues with the deep_iterable anymore using the attrs master and mypy 0.730

Maybe they fixed the Optional issue.

[hynek]

Can we close then?

[euresti]

Sure!

[ojii]

is there a rough schedule for the next release of attrs? (I just want to know if I need to ship my workarounds or can wait a little bit for an attrs release, sorry if this sounds pushy)

[hynek]

As soon as #580 is merged. Dunno when someone finds time for review. ¯\_(ツ)_/¯

[ojii]

finally got around to update my library and can confirm these typing issues went away. thank you all for your hard work.