Honor return type of __new__

[JukkaL]

Currently the return type of A(...) is implicitly A even if A.__new__ return something else. If the return type is Any, maybe A(...) should have type Any, and if it's a subclass, we should also use that. Currently this is a little difficult to implement due to implementation limitations (we derive the type object identity from the return type of a type object). Maybe we should introduce a new attribute to Callable for this purpose.

[JukkaL]

This is follow-up to #982.

[gvanrossum]

Honestly I don't think that A() should ever return something that's not an instance of A (or a subclass), even though technically __new__() can return anything it likes. Somehow I worry that if __new__() were defined as returning Any for some class and from that we'd infer that A() returned Any we would weaken the type checking -- I'd much rather get an error if the declared return type of __new__() isn't A or a subclass of A.

Have you seen any real code that violates this rule?

[JukkaL]

No, I haven't seen real code that uses this, but I've seen tutorials that mention that this is possible -- they always mention that you probably shouldn't do that.

Okay, what about this:

• If the return type is Any (either implicit or explicit), we don't check return statements within __new__ but the type of A() is still A.
• If the return type of __new__ is not A or a subclass of A (and not Any), reject it as a type error.
• If the return type B is a proper subclass of A, A() has type B.

[gvanrossum]

SGTM

[ilevkivskyi]

@JukkaL On one hand, constructor returning an instance of different class is unacceptable in static typed languages. But on other hand, this feature is sometimes quite useful in Python. Even typing.py uses this for Union and for NamedTuple (since 3.6b1).

In general, it would be nice if mypy minimizes the number of false positives like flagging this code:

class C:
    def __new__(cls):
        return 0

C() + 1

with Unsupported operand types for + ("C" and "int"). Maybe there could be an option to allow this.

[davidroeca]

One thought: the return type of __new__, if explicitly defined and different from the class name itself, could implicitly be the class's supertype.

class ZeroLiteral:  # supertype: int
    def __new__(cls) -> int:
        return 0

This would also be a general alternative to python/typing#415 and could likely handle #3062

[chadrik]

We make use of __new__ to promote base classes to leaf classes in a few places. I don't love it, but it'll take some serious effort to untangle it, so I'm gonna +1 this for the case of base class promoting to subclasses.

Ours works something like this:

from typing import Union

class Base(object):
    def __init__(self, name):
        self.name = name

    def __new__(cls, name):
        # type: (str) -> Union[A, B]
        if cls is Base:
            if name == 'A':
                return A(name)
            else:
                return B(name)
        else:
            # A or B
            return object.__new__(cls)

class A(Base):
    pass

class B(Base):
    pass

b = Base('B')
reveal_type(b)

The revealed type is Base, but I expect it to Union[A, B].

Also, this seems related to #3307.

[nierob]

Another use case that I had is to have async constructor, poor man's __ainit__:

class WorkItem(metaclass=ABCMeta):
    async def __new__(cls, *args, **kwargs):
        """ Override __new__ to support our custom `async` constructors. """
        o = super().__new__(cls)
        await o.ainit(*args, **kwargs)
        return o

That actually fails in a different place, as in this case new returns the right thing, but

class BuildItem(WorkItem):    ...
toolsetItem = await BuildItem(...)

fails with:

error: Incompatible types in "await" (actual type "BuildItem", expected type "Awaitable[Any]")

[ilevkivskyi]

Another (less controversial) use case is at least to honor the return type of __new__ for generics, see #4236 (comment)

[achimnol]

Maybe #7477 would be another motivating example for this. After applying my aobject conecpt around here and there I also encountered this issue afterwards. (I thought I succesfully worked around with __ainit__(), but my aobject package were not being type-checked in the project I've been working on due to missing py.typed marker.)

I've tried the latest master which contains #7188 but await SomeAObjectVariant() still shows the following error:

Incompatible types in "await" (actual type "SomeAObjectVariant", expected type "Awaitable[Any]")

Defining a copy of __new__() as new() worked fine in both type checks and the runtime, using it as await SomeAObjectVariant.new().

[jbrockmendel]

Have you seen any real code that violates this rule?

pandas.Timestamp, pandas.Timedelta, and pandas.Period can all return pd.NaT, which doesn't subclass any of the three. xref pandas-dev/pandas#40766

[moi90]

Have you seen any real code that violates this rule?

@gvanrossum I have a use case for this: In my library MorphoCut, a computational graph is constructed of individual processing nodes. For the readability of the code, it would be really great to instantiate a node but getting something else instead of the instance object.

Here is a coarse sketch:

Expand

from typing import Dict, Iterable, List, TYPE_CHECKING, Type

# When instantiated, nodes are put into this list
nodes: List["Node"] = []


class Variable:
    """A variable is a placeholder for a value in the stream."""


class Node:
    """A Node processes stream objects."""

    # Error: Incompatible return type for "__new__" (returns "Variable", but must return a subtype of "Node")
    def __new__(cls: Type["Node"], *args, **kwargs) -> Variable:
        print(f"{cls.__name__}.__new__")
        node: Node = object.__new__(cls)

        # Initialize
        node.__init__(*args, **kwargs)

        return node.output

    def __init__(self) -> None:
        print(f"{self.__class__.__name__}.__init__")
        self.output = Variable()

        nodes.append(self)

    def transform_stream(self, stream: Iterable):
        for obj in stream:
            obj[self.output] = self.transform(obj)
            yield obj

    def transform(self, obj):
        raise NotImplementedError()


class Range(Node):
    """A Node that produces numbers in a range."""

    def __init__(self, stop) -> None:
        super().__init__()

        self.stop = stop

    def transform_stream(self, stream: Iterable):
        for i in range(self.stop):
            yield {self.output: i}


class Incr(Node):
    """A Node that increments the incoming value."""

    def __init__(self, x: Variable) -> None:
        super().__init__()

        self.x = x

    def transform(self, obj):
        return obj[self.x] + 1


# Build the pipeline. (This is what a user of the library does. Therefore, it needs to be maximally easy to see what is going on.)

value = Range(10)

if TYPE_CHECKING:
    reveal_type(value)  # Expected: Variable. Actually: Source.

# Error: Argument 1 to "Incr" has incompatible type "Range"; expected "Variable"
value2 = Incr(value)


###

# Build the computational graph by stacking the individual Node.transform_stream

stream: List[Dict] = [{}]
for node in nodes:
    stream = node.transform_stream(stream)

# Execute the computational graph
for obj in stream:
    print(obj)

(In reality, it's still a bit more difficult.)

There are other ways of doing this, but all (that I came up with so far) have their downsides. (Class decorators are equally badly supported. Instantiating, then retrieving the output property harms brevity and readability.)

[eagleoflqj]

Have you seen any real code that violates this rule?

SymPy.
To represent an unevaluated object like sin(x), the sin must be defined as a class rather than a function. But if you construct sin(0), it's automatically evaluated to a sympy Integer 0, which is not an instance of Application (the base class of sin that calls __new__).

[gvanrossum]

To represent an unevaluated object like sin(x), the sin must be defined as a class rather than a function.

Nah, it could be a function returning a class instance. I trust there's a reason, but it's probably a little more complicated than that. :-)

But if you construct sin(0), it's automatically evaluated to a sympy Integer 0, which is not an instance of Application (the base class of sin that calls __new__).

Yeah, that's a good example actually.

[moi90]

To represent an unevaluated object like sin(x), the sin must be defined as a class rather than a function.

Nah, it could be a function returning a class instance. I trust there's a reason, but it's probably a little more complicated than that. :-)

If you use a function returning a class instance, you have to replicate the parameters of the constructor (if you want a proper signature / proper typing). I find this very inconvenient.

[twoertwein]

Maybe an example where it might be reasonable for __new__ to not return the class instance.

test.pyi:

from typing import (
    Iterable,
    NoReturn,
    overload,
)

class A:
    @overload
    def __new__(cls, x: str | bytes) -> NoReturn: ...  # overlapping, but works in practise as type checkers pick the first matching overload
    @overload
    def __new__(cls, x: Iterable) -> A: ...

reveal_type(A([]))
reveal_type(A("a"))  # mypy: DataFrame, pyright: NoReturn

The above would help to tightly type pd.DataFrame.

[sdeframond]

Honestly I don't think that A() should ever return something that's not an instance of A (or a subclass)

It depends on the purpose of Mypy:

• Is it to enable people to do whateveer they want, in a type-safe way ?
• Or is it to enforce good practices ?

I believe that the later would better be served by a linter than by a type-checker.

[oscarbenjamin]

SymPy.
To represent an unevaluated object like sin(x), the sin must be defined as a class rather than a function. But if you construct sin(0), it's automatically evaluated to a sympy Integer 0, which is not an instance of Application (the base class of sin that calls __new__).

I have been working adding type hints for this to sympy (sympy/sympy#25103) but hitting up against this issue. To be clear I think that SymPy's design here is not great and ideally expression heads would not be classes. Handling typing in a symbolic context is very difficult though and I'm still not sure what good ways to do this are (the Julia symbolics.jl folks are also struggling with the same thing). We are talking about it but changing this in sympy at this stage would be extremely difficult (there are hundreds of these classes spreading over ~100k lines of code and then more in downstream code as well).

I just want to note that pyright handles this differently and behaves exactly in the way that I would expect:

from __future__ import annotations

class A:
    def __new__(cls) -> A:
        return super().__new__(cls)

class B(A):
    def __new__(cls) -> A:
        return super().__new__(cls)

reveal_type(B())

$ mypy q.py
q.py:8: error: Incompatible return type for "__new__" (returns "A", but must return a subtype of "B")  [misc]
q.py:11: note: Revealed type is "q.B"
Found 1 error in 1 file (checked 1 source file)
$ pyright q.py
...
  ./q.py:11:13 - information: Type of "B()" is "A"
0 errors, 0 warnings, 1 information

I don't mind adding type: ignore for the "error" that mypy reports but there is no way to get mypy to understand that B() returns type A which ultimately means that mypy will reject valid downstream/user code that I don't control. In other words:

1. I don't necessarily object to mypy reporting the return hint as an error.
2. I think that mypy should still respect that hint for inference though (the hint is accurate regardless of whether anyone dislikes it).
3. It would be nice if mypy and pyright were consistent about inference especially when there are explicit hints.

[nickdrozd]

I have a real-world case where __new__ can return a different type. I am working with algebraic expressions. These expressions are int-like and indeed can be evaluated to ints, but that evaluation can be expensive (large numbers involved), so we keep them unevaluated for as long as possible. Algebraic expression and actual int numbers are commingled and interact with each other.

When building an expression, I want to check for trivial values and abandon the expression if trivial values are found. This can be done in __new__. Here is some toy example code:

from __future__ import annotations

class Add:
    l: IntLike
    r: IntLike

    def __new__(cls, l: IntLike, r: IntLike) -> IntLike:  # type: ignore[misc]
        return (
            l if r == 0 else
            r if l == 0 else
            super().__new__(cls)
        )

    def __init__(self, l: IntLike, r: IntLike):
        self.l = l
        self.r = r

    def __repr__(self) -> str:
        return f'({self.l} + {self.r})'

    def __add__(self, other: IntLike) -> IntLike:
        return Add(self, other)

IntLike = int | Add

So for example, Add(1, 5) produces an Add object, but Add(0, 5) just produces the int 5.

This runs just fine, and is perfectly sensible AFAICT. So to me it seems like Mypy's warning here is an opinionated complaint about style rather than an error being flagged. Maybe there could be a flag for allowing / forbidding this?

Note that the type of __new__ is explicitly declared, so there is no Any-related confusion.

[zmievsa]

My library, cached_classproperty uses this behavior to specify that a cached_classproperty returns a constant from its __new__ because unlike a regular cached_property, cached_classproperty will also get executed when accessed from a class.

Are we planning to do anything about this task? Yes, simplicity is important but people have given so many use cases here. Pyright supports this feature too.

[Booplicate]

• Is it to enable people to do whateveer they want, in a type-safe way ?

Those two don't go along together. You either use static typing and follow the rules or make your code dynamic. You can't do "whatever you want" in a statically typed world. When you change the well defined signature, it's already unsafe because by nature people will make assumptions that your code follows the common practices/guidelines/rules. Your code lies about what it does and you want mypy to hide it.

Most of the examples can be replaced with a more appropriate - even if not the prettiest, but definitely less cursed - construct.

[oscarbenjamin]

When you change the well defined signature, it's already unsafe because by nature people will make assumptions that your code follows the common practices/guidelines/rules.

Can you point to anything that defines what the "rules" are for __new__?

The docs are clear that cls.__new__ may return an object that is not an instance of cls:
https://docs.python.org/3/reference/datamodel.html#object.__new__

The question as correctly asked above is whether mypy is supposed to be a type checker or a linter. Personally I would rather keep linting out of type checking and I definitely don't want opinionated lint rules to be baked into type inference.

Note that returning a different type here does not mean that the types are completely undefined. In the SymPy case there is a class Expr which has many subclasses like sin, Integer etc. The invariant is that if A is a subclass of Expr then A(...) will return an instance of Expr but not necessarily an instance of A. So sin(1) returns an instance of sin and sin(0) returns an instance of Integer, but always sin(anything) returns an instance of Expr because both sin and Integer (and hundreds of other classes) are subclasses of Expr.

It is not difficult to represent what __new__ does in type inference. It is just a class method that is called by type.__call__ which returns the same object:

class type:
    def __call__(cls, *args, **kwargs):
        obj = cls.__new__(cls, *args, **kwargs)
        if isinstance(obj, cls):
            obj.__init__(*args, **kwargs)
        return obj

Having looked a little through the mypy code that handles this I imagine that the mypy code could be both simpler and more robust if it could just represent type.__call__ in this way. Then mypy could apply ordinary type checking rules to __new__ as a class method rather than having many ad-hoc rules for __new__ and __init__ (and for metaclasses).

In fact if I just use a class method new instead of __new__ then mypy can understand what new returns just fine but still fails to understand __new__:

class Expr:
    def __new__(cls, *args) -> 'Expr':
        return cls.new(*args)

    @classmethod
    def new(cls, *args) -> 'Expr':
        return object.__new__(cls)

class sin(Expr):
    @classmethod
    def new(cls, arg) -> Expr:
        if arg == 0:
            return Integer(0)
        else:
            return object.__new__(cls)

class Integer(Expr):
    @classmethod
    def new(cls, arg) -> Expr:
        return object.__new__(cls)

exprs = [sin(0), sin(1), Integer(0), Integer(1)]
assert all(isinstance(expr, Expr) for expr in exprs)

reveal_type(sin.new(0)) # Expr (correct)
reveal_type(sin(0))     # sin (incorrect)

Here pyright understands the type of sin(0) but mypy gets it wrong:

$ mypy t.py
t.py:25: note: Revealed type is "t.Expr"
t.py:26: note: Revealed type is "t.sin"
Success: no issues found in 1 source file
$ pyright t.py
./t.py:25:13 - information: Type of "sin.new(0)" is "Expr"
 ./t.py:26:13 - information: Type of "sin(0)" is "Expr"
0 errors, 0 warnings, 2 informations ```

The fact that mypy can understand .new here but not .__new__ shows that this is because of ad-hoc special casing of __new__ rather than any limitation of mypy's general type inference capability. The fact that pyright can infer the type correctly in both cases shows that there is no reason why a type checker should not be able to do this.

The rejection of returning other types from cls.__new__ is just an arbitrary rule here:

mypy/mypy/checker.py

Lines 1535 to 1543 in 5b1a231

	# And that it returns a subtype of the class
	self.check_subtype(
	bound_type.ret_type,
	self_type,
	fdef,
	message_registry.INVALID_NEW_TYPE,
	"returns",
	"but must return a subtype of",
	)

The inference in mypy bakes in the assumption that cls.__new__ returns an instance of cls e.g.:

mypy/mypy/checkmember.py

Lines 1244 to 1312 in 5b1a231

	def type_object_type(info: TypeInfo, named_type: Callable[[str], Instance]) -> ProperType:
	"""Return the type of a type object.
	For a generic type G with type variables T and S the type is generally of form
	Callable[..., G[T, S]]
	where ... are argument types for the __init__/__new__ method (without the self
	argument). Also, the fallback type will be 'type' instead of 'function'.
	"""
	# We take the type from whichever of __init__ and __new__ is first
	# in the MRO, preferring __init__ if there is a tie.
	init_method = info.get("__init__")
	new_method = info.get("__new__")
	if not init_method or not is_valid_constructor(init_method.node):
	# Must be an invalid class definition.
	return AnyType(TypeOfAny.from_error)
	# There *should* always be a __new__ method except the test stubs
	# lack it, so just copy init_method in that situation
	new_method = new_method or init_method
	if not is_valid_constructor(new_method.node):
	# Must be an invalid class definition.
	return AnyType(TypeOfAny.from_error)
	# The two is_valid_constructor() checks ensure this.
	assert isinstance(new_method.node, (SYMBOL_FUNCBASE_TYPES, Decorator))
	assert isinstance(init_method.node, (SYMBOL_FUNCBASE_TYPES, Decorator))
	init_index = info.mro.index(init_method.node.info)
	new_index = info.mro.index(new_method.node.info)
	fallback = info.metaclass_type or named_type("builtins.type")
	if init_index < new_index:
	method: FuncBase | Decorator = init_method.node
	is_new = False
	elif init_index > new_index:
	method = new_method.node
	is_new = True
	else:
	if init_method.node.info.fullname == "builtins.object":
	# Both are defined by object. But if we've got a bogus
	# base class, we can't know for sure, so check for that.
	if info.fallback_to_any:
	# Construct a universal callable as the prototype.
	any_type = AnyType(TypeOfAny.special_form)
	sig = CallableType(
	arg_types=[any_type, any_type],
	arg_kinds=[ARG_STAR, ARG_STAR2],
	arg_names=["_args", "_kwds"],
	ret_type=any_type,
	fallback=named_type("builtins.function"),
	)
	return class_callable(sig, info, fallback, None, is_new=False)
	# Otherwise prefer __init__ in a tie. It isn't clear that this
	# is the right thing, but __new__ caused problems with
	# typeshed (#5647).
	method = init_method.node
	is_new = False
	# Construct callable type based on signature of __init__. Adjust
	# return type and insert type arguments.
	if isinstance(method, FuncBase):
	t = function_type(method, fallback)
	else:
	assert isinstance(method.type, ProperType)
	assert isinstance(method.type, FunctionLike) # is_valid_constructor() ensures this
	t = method.type
	return type_object_type_from_function(t, info, method.info, fallback, is_new)

The comment in the code

We take the type from whichever of init and new is first in the MRO, preferring init if there is a tie.

bears no relation to the actual semantics of __init__ and __new__ as used in type.__call__ (there is no MRO combining these methods). The actual semantics are not complicated:

1. A.__new__ is called and returns an instance b of type B.
2. If b is an instance of A then b.__init__ (i.e. B.__init__) is called with the same arguments.

Note that A.__init__ may not be called at all depending on the type of what is returned by A.__new__ regardless of where in the MRO __new__ and __init__ are: __new__ always precedes __init__.

[Booplicate]

Can you point to anything that defines what the "rules" are for __new__?

I believe the returned object must be of the same type. It also doesn't seem to be a coroutine function.

I do see this

which implies it may return anything. Or it could imply that __init__ is being called only after successful __new__?

The invariant is that if A is a subclass of Expr then A(...) will return an instance of Expr but not necessarily an instance of A.

In my mind if you want to create an instance of A, you'd use A.__new__, if you want an instance of Expr, then you'd use Expr.__new__, it reads naturally, does it not? If we're in a situation where A.__new__ may not create an instance of A, I'd call such code unsafe because we end up with something unexpected. Maybe I fail to see something. What makes that code valid?

The question as correctly asked above is whether mypy is supposed to be a type checker or a linter

I see it as a correct type check, not as a lint. Since that method is tied to that class, then it should return instances of the type. However, if the docs are misleading and __new__ actually returns Any and what you found

this is because of ad-hoc special casing of __new__

is true, then you are correct. I wonder why mypy had that special case in the first place, has __new__ signature been changed?

[oscarbenjamin]

I wonder why mypy had that special case in the first place, has __new__ signature been changed?

Nothing has changed except mypy making arbitrary decisions. The __new__ class method has been around a long time and one of the things that it has long been used for is to turn type.__call__ into a factory function that can return instances of other types. The fact that it can be used in this way is the reason that it is documented that __init__ will not be called if __new__ returns an instance of a different type (whose __init__ would likely have been called already but with different arguments).

The signature of __new__ has not changed because in fact nothing has changed about __new__ for a long time and its behaviour and usage predate all of typing and type hints in Python. Now that type signatures are a thing it has become possible to add type annotations for __new__ like:

class A:
    def __new__(cls) -> B:
        return B()

However mypy has made two opinionated decisions about this that are not based on any previously documented rules or the actual behaviour of Python at runtime:

• mypy reports an error for the A.__new__ method if the return type is not a subclass of A (mypy rejects valid code)
• mypy assumes that A(...) will always return an instance of A in inference (mypy infers the wrong type).

I don't mind mypy reporting an error at __new__ if I can just add type: ignore there. The incorrect inference is the real problem here because it makes all the type checking wrong for any downstream users who are trying to use e.g. SymPy and mypy together and there is no way for me to add anything like type: ignore to make it work correctly.

The fact is that A(...) returns an instance of B as clearly stated in the type annotation for A.__new__. For mypy to infer anything else is just erroneous. Note that in my example code for Expr above mypy as a type checker accepts all of the code without any errors but then gets the inference for sin(0) wrong.

[nickdrozd]

When you change the well defined signature, it's already unsafe because by nature people will make assumptions that your code follows the common practices/guidelines/rules.

A caller making incorrect assumptions about a return type is exactly the sort of thing that a typechecker ought to prevent, except that in this case the typechecker itself also incorrectly assumes the return type.

Here's an example:

from __future__ import annotations

from random import randint
from typing import TYPE_CHECKING

class A:
    def __new__(cls) -> int | A:  # type: ignore[misc]
        if randint(0, 1):
            return object.__new__(cls)
        else:
            return 5

def f(x: A) -> None:
    assert isinstance(x, A)

a = A()

if TYPE_CHECKING:
    reveal_type(a)
else:
    print(a)

f(a)

This code will fail half the time. Mypy ought to be able to figure out why, but because it disregards the clearly specified signature of A.__new__, it can't see the error.

As @oscarbenjamin points out, this is a full-blown inference bug. Maybe the issue should be reopened?