ABCMeta.register support

[miedzinski]

class A(metaclass=abc.ABCMeta): pass
class B: pass
A.register(B)
a: A = B()  # currently E: Incompatible types in assignment (expression has type "B", variable has type "A")
print(isinstance(a, A))  # -> True

Mypy could support cases like this by special casing ABCMeta.register calls. __subclasshook__, __subclasscheck__, and __instancecheck__ are more troublesome and could be left ignored.

[JukkaL]

This was on my radar early in mypy development. I didn't work on this back then because this is a tricky feature. For example, if module x has A.register(B), it can affect a seemingly unrelated module y if it tries to use B as a subclass of A. This is 'non-modular' behavior, and mypy would somehow have to figure out that y depends on x even if it doesn't import it, even indirectly. Currently dependencies between modules are completely described by the import graph.

There are few potential ways to support this:

1. Have A.register(B) invalidate all modules that have been already type checked and that may use A and B. This can be tricky to implement and could slow down things, as we may have to process modules multiple times. Conservatively, A.register(B) would invalidate every module that uses A or B or a subclass of B.
2. Like (1), but also cache all register calls from the previous run and somehow use these to tweak the order of processing modules. The goal is to avoid reprocessing modules multiple times. It's unclear if this idea would work.
3. Make A.register(B) visible/active only in modules that (recursively) import the module that has this declaration. Mypy would have to keep track of these register calls defined in each module and which of them are active in any given module. This would deviate from Python semantics and could be a little confusing. However, there would no 'spooky action at a distance'.
4. Have a new pass that only finds A.register(B) declarations in the entire program. This would get run before type checking. We can probably make this reasonably fast in incremental mode by being clever with caching etc. [This would probably result in mypy having to keep the entire program in memory when doing a non-incremental run, which would be unfortunate. Currently we could strip most nodes in ASTs of already processed modules to lower RAM use.]
5. Only respect those register calls that mypy has seen at any given stage of processing. This would approximate how these work at runtime, but since mypy can process modules in a different order from Python, this wouldn't always work as expected and could be confusing. This would be easy to implement, at least. We'd probably want to process all register calls inside a module before processing the rest of the module, though, as otherwise the order of definitions within a module would affect type checking results, which would be confusing.

[JukkaL]

It seems likely that mypy will support structural subtyping before it supports registering ABCs. Structural subtyping/protocols can be used for a similar effect, but that should be easier to implement. See python/typing#11 for more information.

[ddfisher]

Alternatively, instead of doing something very complicated to make sure A.register(B) works everywhere, just make A.register(B) a type error in modules that don't define either A or B, because you really shouldn't be doing that. (And it's pretty common to disallow such things, see for example Rust's rules against orphaned impls.) Solution 3 also seems potentially reasonable. Spooky action at a distance rules out all the other potential solutions IMO.

[pkch]

I just ran into a use case where support for A.register(B) would be the only way to make things work:

from typing import *
class A('Iterable[A]'):
    _a: 'List[A]'

    def __iter__(self) -> 'Iterator[A]':
        return iter(self._a)

a = A()
reveal_type(a)  # Revealed type is 'test.A'
for x in a:
    reveal_type(x)  # Revealed type is 'test.A*'

mypy is happy with this - surprisingly given that we don't support recursive types yet (#731). Of course, python runtime isn't too happy that we inherit from a string literal.

The basic problem is that we hacked the class inheritance syntax to do typing-related stuff. But this means we are limited in what we can do - and in particular, we can't use forward declarations there (as well as other constructs that would mess up inheritance; at present, I'm not aware of any specific situations where this comes up).

Since forwarding syntax is already set in stone, the only solution I see is to make this work:

class A:
    _a: 'List[A]'
    def __iter__(self) -> 'Iterator[A]':
        return iter(self._a)
Iterable[A].register(A)```

It looks a little ugly, but at least it's logical.

True, this particular use case will disappear once Protocols (#3132) is merged. But it won't fully solve the problem because sometimes the user might want to rely on nominal rather than structural typing.

Incidentally, this situation already appears in builtins.pyi:

class str(Sequence[str]):
    # ...

It works there because it's a stub, not the original source code, so python runtime doesn't need to look at it.

(Btw, we should probably make mypy catch inheritance from a string literal before the runtime complains.)

[ilevkivskyi]

@pkch

class A(Iterable['A']):
    ...

works both in mypy and at runtime.

Btw, what are the cases where you think people will want nominal checks instead of protocols?

[pkch]

Duh 😆

Nominal checks: if the method naming is too generic (validate, save, etc.) so the user has (or is afraid to have) a bunch of unrelated classes that would look compatible based on purely structural checks.