Conformance: class constructor to callable conversion doesn't fully work

[migeed-z]

Describe the Bug

The following code should fully type check (unittest can be found in pyrefly/pyrefly/lib/test/callable.rs)

from typing import Callable, ParamSpec, TypeVar, Self, assert_type, overload, Generic

P = ParamSpec("P")
R = TypeVar("R")
T = TypeVar("T")

def accepts_callable(cb: Callable[P, R]) -> Callable[P, R]:
    return cb

class Class3:
    def __new__(cls, *args, **kwargs) -> Self: ...
    def __init__(self, x: int) -> None: ...

r3 = accepts_callable(Class3)

class Class7(Generic[T]):
    @overload
    def __init__(self: "Class7[int]", x: int) -> None: ...
    @overload
    def __init__(self: "Class7[str]", x: str) -> None: ...
    def __init__(self, x: int | str) -> None:
        pass

r7 = accepts_callable(Class7)
# pyrefly incorrectly errors on these - should be OK
assert_type(r7(""), Class7[str])  # E: assert_type(Class7[int], Class7[str]) failed 

class Class8(Generic[T]):
    def __new__(cls, x: list[T], y: list[T]) -> Self:
        return super().__new__(cls)

r8 = accepts_callable(Class8)
# pyrefly incorrectly errors on this - should be OK
assert_type(r8([""], [""]), Class8[str])  # E: assert_type(Class8[Any], Class8[str]) failed

Sandbox Link

https://pyrefly.org/sandbox/?project=N4IgZglgNgpgziAXKOBDAdgEwEYHsAeAdAA4CeS4ATrgLYAEALqcROgOZ0Q3G6UN0BhVFCipssADR0ACqkqoaAZWIwAxlIAqzGADU5UxTChgpqOHBh8A%2BkxVTcAN0tRcqTFIDiMdJYiqAOuiB0nQAvDJyCspqABT%2BINLxAJSBAEphdFoqepRxIKnJgRoZWbpyeRqFQVgwYHSoqqowxAxwVqrCouIwMarYiIKdYrAA2tJSqQC6SXQAtAB8gyLDMGMTk4iBdNt0lDAMAK6U6HR9gYGqouaCV3AAzJsnO5i1dFZWPgDu771QcFIAKjkbH%2BdABAIA1p9gXAZgs6IZjANCCits9Xu9WBAGD8LMYpPgBqwGHDFgA5XA%2BZGo6qUO4ZBpNFptDrLboxAS3O4paqXMxwG78gDsMS8PkofhGGmmjx2dAAAo5nK5MGjti86pj0NjcUYwAN4pzhSNiZN4gSiegSXNyZSYNTCGqFUrKC43E6NW8rFicVYYnj9XRDbchSM4AxKGaQBa6OHKKS6BSqXQUY6nuqMd7tb7-XqY8S6AAfWMRhNJ%2B1OuXEfnndCUIUMxrNVrtIbso3mIU8gDEdDIezAUFInHQql4e1UDCHdEs1EoAspjAAFvAYHNY0vcAcoJg6Ng1wB5ADSgX5lhxth69byySkHbgobj022vYAogMz9ZLxyQyarZM71-J8ZjAVBoBgXdaz5a57wADlFbxfFUKUZQ9TMvh%2BS5QUJOgoAgcMUKkUgBjwgjpQTRF9UrHY9kOY5YwOFRciSQh3gwv0sJ5QJKFgxsmRbVkulgH9%2BVgns%2B1IAdp1YMdKAnKdh1nXgFxOBgl3w9c4E3bdd33Ohj1PcxzxsbQYh4mIRnieIALoSyQGspJANEsMI2fOg3w-Iyv1MuCRgAQXQUgbN84C6FA8DMGjEADgYaA4BIchEBAXsAFVYrwpgwoOUdYspOBa09MBeBoVAcXQA4aH3XIcOJBM41lGj9iOE4wHiMkKqqgZgHwABfKz0Ci-taiHQgGFoKAKF7aRJOG4c0CwPB8FOSlIDYI5SogSk017Qw1yXBgGGIOBEAAehOobB1IQheDYE7vBOzBcFUOATrHdBVvW3L0BOsLeHqBwwKEtc3o%2B%2BQvroXAWk29B4sCMg1MpWYnHnaGMniO5CAARgAJn6kAeokEAGlipwADFwIoeacAIBL8aAA

(Only applicable for extension issues) IDE Information

No response

[fangyi-zhou]

I'd suggest to split this one into:

• handling __new__ with generics
• Figure out why overloads get lost after passing through accepts_callable (See test case in Handle self type annotation when converting __init__ as a callable #2226 -- constructor to callable conversion should be already correct)

[OutSquareCapital]

I have an issue that may be related, basedpyright has no problem resolving this, but neither pyrefly nor ty can.

Reproducible example

from collections.abc import Callable, Iterable, Iterator
from typing import override, reveal_type


class MyIterator[T](Iterator[T]):
    def __init__(self, data: Iterable[T]) -> None:  # noqa: D107
        self._inner: Iterator[T] = iter(data)

    @override
    def __iter__(self) -> Iterator[T]:  # noqa: PYI034
        return self._inner

    @override
    def __next__(self) -> T:
        return next(self._inner)

    def collect[R](self, collector: Callable[[Iterator[T]], R] = list[T]) -> R:
        return collector(self._inner)


x = MyIterator([1, 2, 3])
reveal_type(x)
y = x.collect()
reveal_type(y)
z = x.collect(set)
reveal_type(z)

Output

basedpyright

PS C:\Users\stett\Documents\python\pyochain> uv run basedpyright t.py
c:\Users\stett\Documents\python\pyochain\t.py
  c:\Users\stett\Documents\python\pyochain\t.py:22:13 - information: Type of "x" is "MyIterator[int]"
  c:\Users\stett\Documents\python\pyochain\t.py:24:13 - information: Type of "y" is "list[int]"
  c:\Users\stett\Documents\python\pyochain\t.py:26:13 - information: Type of "z" is "set[int]"
0 errors, 0 warnings, 3 notes

ty

PS C:\Users\stett\Documents\python\pyochain> uvx ty check t.py       
error[invalid-parameter-default]: Default value of type `<class 'list[T@MyIterator]'>` is not assignable to annotated parameter type `(Iterator[T@MyIterator], /) -> R@collect`
  --> t.py:17:26
   |
17 |     def collect[R](self, collector: Callable[[Iterator[T]], R] = list[T]) -> R:
   |                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |

info[revealed-type]: Revealed type
  --> t.py:22:13
   |
22 | reveal_type(x)
   |             ^ `MyIterator[int]`
   |

info[revealed-type]: Revealed type
  --> t.py:24:13
   |
24 | reveal_type(y)
   |             ^ `Unknown`
   |

info[revealed-type]: Revealed type
  --> t.py:26:13
   |
26 | reveal_type(z)
   |             ^ `Unknown`
   |

Found 4 diagnostics

pyrefly

PS C:\Users\stett\Documents\python\pyochain> uvx pyrefly check t.py  
ERROR Default `type[list[T]]` is not assignable to parameter `collector` with type `(Iterator[T]) -> R` [bad-function-definition]
  --> t.py:17:66
   |
17 |     def collect[R](self, collector: Callable[[Iterator[T]], R] = list[T]) -> R:
   |                                     --------------------------   ^^^^^^^
   |                                     |
   |                                     declared type
   |
 INFO revealed type: MyIterator[int] [reveal-type]
  --> t.py:22:12
   |
22 | reveal_type(x)
   |            ---
   |
 INFO revealed type: Unknown [reveal-type]
  --> t.py:24:12
   |
24 | reveal_type(y)
   |            ---
   |
 INFO revealed type: set[Unknown] [reveal-type]
  --> t.py:26:12
   |
26 | reveal_type(z)
   |            ---
   |
 INFO 1 error
PS C:\Users\stett\Documents\python\pyochain> 

[fangyi-zhou]

Currently, only Class8 in the original snippet is failure. I digged a bit deeper, looks like the generic parameters are lost when processing class methods in general

Setup:

def accepts_callable(cb: Callable[P, R]) -> Callable[P, R]:
    return cb

class Box(Generic[T]):
    pass

Doesn't work (becomes "Unknown"):

class Factory(Generic[T]):
    @classmethod
    def make(cls, x: list[T], y: list[T]) -> Box[T]: ...

r = accepts_callable(Factory.make)
reveal_type(r)  # pyrefly: (x: list[Unknown], y: list[Unknown]) -> Box[Unknown]
assert_type(r([""], [""]), Box[str])  # currently fails

Works (becomes ForAll):

def make_function[T](x: list[T], y: list[T]) -> Box[T]: ...

r2 = accepts_callable(make_function)
reveal_type(r2)  # pyrefly: [T](x: list[T], y: list[T]) -> Box[T]
assert_type(r2([""], [""]), Box[str])

https://pyrefly.org/sandbox/?project=N4IgZglgNgpgziAXKOBDAdgEwEYHsAeAdAA4CeS4ATrgLYAEALqcROgOZ0Q3G6UN0BhVFCipssADR0A4jHQxKEAMZSACqkqoaAZWIwVdACrMYANQ1TUcOAoYB9JnqmUYANxjCHJgDrpfqugBeOnVNHT0lAApvEFUYgEpfACUgoxNzSmiQJITfQ1TjPQysw1y-LBgwOlQlJRhiBjg7JWFRcRhIpWxEQVaxWABtVSkkgF14ugBaAD5ekX6YIZHRxF86dboXBgBXSnQ6Lt9fJVFrOgAhAkjZeUUlAcNx1f2N4is4I-QT97oAMRqGLxSNc5AplA8nmsNgABb7WGgwBgAC1wmCh60wlToNFQAGsOic4FJ8D0oBA4AwIVJSKTyZTHhMZhcCBCeoR2Z9KKkanUGk0WvN2pF-kpAZRSIQcfjEugXO5PI4OpQJnQAMR0MguMBQGl0SIkuhkikDACq6Fx6FwAHd0KNqbTjWaLdbbYzZpd8KbzZabaNPu9bF49JFMgMYjE7XQwyAI-EpB6BhTKON1uqlLsXOgGDq6GBUNAPuVfJiqlKYHYwNsvgwILh0BD9Q76ZHdUbm27mZ7HmyOeVKAAmbm1eqNZp9IVlitV0W19AyuUeKBBpX9lXqzWVHU9BsGttUuitukQjsJx7%2B6yBxUh-uRaMRqR3kDjeMspPjXwgCQgbY1qBwEjkIgIDqiav4QEwubTjWdaFsWWJgLwOL2Og2w0NgCiNpwWYdkmzwbJsiK7PsYAxAAcqh6GUD0wD4AAvuG6CfiAG7ahKgI0FAFDqqopBajmaBYHg%2BAHHWkBsLsqDQeghC%2BOq2gwDAdBIgwDDEHAiAAPQaSxOqELwbAaXIGmYLgShwBpSiiRA4maFJGm5rw1SuPmbSwCJ6BiRJUl0LgDSzv%2BvhkMidaTO4lBwLOqQxAAzIQACM-YMSAtFfgCEDuL80AwBQAk4AQAHJUAA