TYP: Mixed scalar/int binary operations have incorrect type annotation

[oscarbenjamin]

Describe the issue:

The type annotations used for scalar/int binary operations like np.float32(1) * 2 imply that the scalar types are not closed under e.g. multiplication with int:

reveal_type(np.int8(1)) # signedinteger[_8Bit]
reveal_type(np.int8(1) * np.int8(1)) # signedinteger[_8Bit]
reveal_type(np.int8(1) * 1) # signedinteger[_8Bit] | signedinteger[_32Bit | _64Bit]

As far as I can tell mixed operations with int don't actually promote the type:

>>> np.int8(1) * 128
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: Python integer 128 out of bounds for int8

It comes from here:

numpy/numpy/__init__.pyi

Lines 3768 to 3776 in a7eda47

	class signedinteger(integer[_NBit1]):
	def __init__(self, value: _ConvertibleToInt = ..., /) -> None: ...
	__add__: _SignedIntOp[_NBit1]
	__radd__: _SignedIntOp[_NBit1]
	__sub__: _SignedIntOp[_NBit1]
	__rsub__: _SignedIntOp[_NBit1]
	__mul__: _SignedIntOp[_NBit1]
	__rmul__: _SignedIntOp[_NBit1]

And that uses:

numpy/numpy/_typing/_callable.pyi

Lines 207 to 222 in a7eda47

	@type_check_only
	class _SignedIntOp(Protocol[_NBit1]):
	@overload
	def __call__(self, other: bool, /) -> signedinteger[_NBit1]: ...
	@overload
	def __call__(self, other: int, /) -> signedinteger[_NBit1] | int_: ...
	@overload
	def __call__(self, other: float, /) -> floating[_NBit1] | float64: ...
	@overload
	def __call__(
	self, other: complex, /
	) -> complexfloating[_NBit1, _NBit1] | complex128: ...
	@overload
	def __call__(
	self, other: signedinteger[_NBit2], /
	) -> signedinteger[_NBit1] | signedinteger[_NBit2]: ...

I think that the problematic overload is:

def __call__(self, other: int, /) -> signedinteger[_NBit1] | int_: ...

Is there a reason that | int_ is needed there?

Reproduce the code example:

from __future__ import annotations
import numpy as np
from typing import Protocol, Self, reveal_type

class MultiplyWithInt(Protocol):
    def __mul__(self, other: int, /) -> Self:
        ...

a: MultiplyWithInt = 1
b: MultiplyWithInt = 1.0
c: MultiplyWithInt = 1j
d: MultiplyWithInt = np.uint8(1)
e: MultiplyWithInt = np.uint16(1)
f: MultiplyWithInt = np.uint32(1)
g: MultiplyWithInt = np.uint64(1)
h: MultiplyWithInt = np.int8(1) # type check error
i: MultiplyWithInt = np.int16(1) # type check error
j: MultiplyWithInt = np.int32(1) # type check error
k: MultiplyWithInt = np.int64(1)
l: MultiplyWithInt = np.float32(1.0) # type check error
m: MultiplyWithInt = np.float64(1.0)
n: MultiplyWithInt = np.complex64(1) # type check error
o: MultiplyWithInt = np.complex128(1)

reveal_type(np.uint8(1)) # unsignedinteger[_8Bit]
reveal_type(np.uint8(1) * 1) # Any
reveal_type(np.uint8(1) * np.uint8(1)) # unsignedinteger[_8Bit]

reveal_type(np.int8(1)) # signedinteger[_8Bit]
reveal_type(np.int8(1) * 1) # signedinteger[_8Bit] | signedinteger[_32Bit | _64Bit]
reveal_type(np.int8(1) * np.int8(1)) # signedinteger[_8Bit]

Error message:

No response

Python and NumPy Versions:

Python 3.12
NumPy 2.2.1

Runtime Environment:

No response

Context for the issue:

I'm trying to write generically typed code with rings like:

from typing import Protocol, Self, Literal

type _PositiveInteger = Literal[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

class RingElement(Protocol):
    """Elements supporting ring operations."""

    def __pos__(self) -> Self: ...
    def __neg__(self) -> Self: ...
    def __add__(self, other: Self, /) -> Self: ...
    def __mul__(self, other: Self | int, /) -> Self: ...
    def __rmul__(self, other: int, /) -> Self: ...
    def __pow__(self, other: _PositiveInteger, /) -> Self: ...

The allowance for multiplication with int is so that with this protocol you can have code like 2*x + y*2. Both mypy and pyright think that some of numpy's scalar types are incompatible with this protocol because they are not closed under multiplication with int.

[jorenham]

Thanks for reporting this!

Technically speaking, the return type annotation for integer.__add__ isn't incorrect here; it's just unnecessarily broad. But as you demonstrated, this can lead to unexpected behavior, which can be especially annoying in the case of structural typing (i.e. a Protocol).

So to answer your question:

Is there a reason that | int_ is needed there?

No. It should be removed. And I will.

But I suppose that it isn't far-fetched to (incorrectly) assume that np.array(1, np.int8) + np.array(2) would have the same scalar type as np.int8(1) + 2 would -- especially if you also take the fact that np.int(8) + 1.0 is np.float64(3.0) into account.

[jorenham]

For (mostly my own) reference, I tabulated the resulting dtypes of y = xi.__add__(xj) (i the rows, j the columns), with xi and xj numpy or builtin scalars, displayed as y.dtype.str[1:], (on my 64bit linux machine with numpy==2.2.1, i.e. f16 is the
longdouble and c32 the clongdouble).
A _ is used in case the result type is the same as that of xi. A cell is empty if __add__ returned NotImplemented (so + would delegate to xj.__radd__).

_	bool	int	float	complex
b1	_	i8	f8	c16
u1	_	_	f8	c16
u2	_	_	f8	c16
u4	_	_	f8	c16
u8	_	_	f8	c16
i1	_	_	f8	c16
i2	_	_	f8	c16
i4	_	_	f8	c16
i8	_	_	f8	c16
f2	_	_	_	c8
f4	_	_	_	c8
f8	_	_	_	c16
f16	_	_	_	c32
c8	_	_	_	_
c16	_	_	_	_
c32	_	_	_	_

(note that np.float64() + complex() is a complex128, but complex() + np.float64() is just a builtins.complex)

_	u1	u2	u4	u8	i1	i2	i4	i8
b1	u1	u2	u4	u8	i1	i2	i4	i8
u1	_				i2
u2	_	_			i4	i4
u4	_	_	_		i8	i8	i8
u8	_	_	_	_	f8	f8	f8	f8
i1	i2	i4	i8	f8	_
i2	_	i4	i8	f8	_	_
i4	_	_	i8	f8	_	_	_
i8	_	_	_	f8	_	_	_	_
f2	_	f4	f8	f8	_	f4	f8	f8
f4	_	_	f8	f8	_	_	f8	f8
f8	_	_	_	_	_	_	_	_
f16	_	_	_	_	_	_	_	_
c8	_	_	c16	c16	_	_	c16	c16
c16	_	_	_	_	_	_	_	_
c32	_	_	_	_	_	_	_	_

_	f2	f4	f8	f16	c8	c16	c32
b1	f2	f4	f8	f16	c8	c16	c32
u1
u2	f4
u4	f8	f8			c16
u8	f8	f8			c16
i1
i2	f4
i4	f8	f8			c16
i8	f8	f8			c16
f2	_
f4	_	_
f8	_	_	_		c16
f16	_	_	_	_	c32	c32
c8	_	_	c16	c32	_
c16	_	_	_	c32	_	_
c32	_	_	_	_	_	_	_

[oscarbenjamin]

Thanks both.

I also found similar problems with arrays rather than scalars although this time the type check errors are almost for the opposite set of dtypes:

from __future__ import annotations
import numpy as np
from typing import Protocol, Self, reveal_type, Literal

class MultiplyWithInt(Protocol):
    def __mul__(self, other: int, /) -> Self:
        ...

a: MultiplyWithInt = 1
b: MultiplyWithInt = 1.0
c: MultiplyWithInt = 1j
d: MultiplyWithInt = np.array([1], dtype=np.uint8) # type check error
e: MultiplyWithInt = np.array([1], dtype=np.uint16) # type check error
f: MultiplyWithInt = np.array([1], dtype=np.uint32) # type check error
g: MultiplyWithInt = np.array([1], dtype=np.uint64) # type check error
h: MultiplyWithInt = np.array([1], dtype=np.int8)
i: MultiplyWithInt = np.array([1], dtype=np.int16)
j: MultiplyWithInt = np.array([1], dtype=np.int32)
k: MultiplyWithInt = np.array([1], dtype=np.int64)
l: MultiplyWithInt = np.array([1], dtype=np.float32)
m: MultiplyWithInt = np.array([1], dtype=np.float64) # type check error
n: MultiplyWithInt = np.array([1], dtype=np.complex64)
o: MultiplyWithInt = np.array([1], dtype=np.complex128) # type check error

reveal_type(np.array([1], dtype=np.uint8)) # dtype[unsignedinteger[_8bit]]
reveal_type(np.array([1], dtype=np.uint8) * 1) # dtype[signedinteger[Any]]
reveal_type(np.array([1], dtype=np.uint8) * np.uint8(1)) # dtype[unsignedinteger[Any]]

reveal_type(np.array([1], dtype=np.int8)) # dtype[signedinteger[_8bit]]
reveal_type(np.array([1], dtype=np.int8) * 1) # dtype[signedinteger[Any]]
reveal_type(np.array([1], dtype=np.int8) * np.int8(1)) # dtype[signedinteger[Any]]

Should I open that as a new issue?

[jorenham]

Thanks both.

I also found similar problems with arrays rather than scalars although this time the type check errors are almost for the opposite set of dtypes:

from __future__ import annotations
import numpy as np
from typing import Protocol, Self, reveal_type, Literal

class MultiplyWithInt(Protocol):
    def __mul__(self, other: int, /) -> Self:
        ...

a: MultiplyWithInt = 1
b: MultiplyWithInt = 1.0
c: MultiplyWithInt = 1j
d: MultiplyWithInt = np.array([1], dtype=np.uint8) # type check error
e: MultiplyWithInt = np.array([1], dtype=np.uint16) # type check error
f: MultiplyWithInt = np.array([1], dtype=np.uint32) # type check error
g: MultiplyWithInt = np.array([1], dtype=np.uint64) # type check error
h: MultiplyWithInt = np.array([1], dtype=np.int8)
i: MultiplyWithInt = np.array([1], dtype=np.int16)
j: MultiplyWithInt = np.array([1], dtype=np.int32)
k: MultiplyWithInt = np.array([1], dtype=np.int64)
l: MultiplyWithInt = np.array([1], dtype=np.float32)
m: MultiplyWithInt = np.array([1], dtype=np.float64) # type check error
n: MultiplyWithInt = np.array([1], dtype=np.complex64)
o: MultiplyWithInt = np.array([1], dtype=np.complex128) # type check error

reveal_type(np.array([1], dtype=np.uint8)) # dtype[unsignedinteger[_8bit]]
reveal_type(np.array([1], dtype=np.uint8) * 1) # dtype[signedinteger[Any]]
reveal_type(np.array([1], dtype=np.uint8) * np.uint8(1)) # dtype[unsignedinteger[Any]]

reveal_type(np.array([1], dtype=np.int8)) # dtype[signedinteger[_8bit]]
reveal_type(np.array([1], dtype=np.int8) * 1) # dtype[signedinteger[Any]]
reveal_type(np.array([1], dtype=np.int8) * np.int8(1)) # dtype[signedinteger[Any]]

Should I open that as a new issue?

Good catch! That indeed sounds like a different issue, and I'll have a look at what's going on there. You're of course also free to try and take a shot at fixing it yourself :)

[jorenham]

Thanks both.

I also found similar problems with arrays rather than scalars although this time the type check errors are almost for the opposite set of dtypes:

from __future__ import annotations
import numpy as np
from typing import Protocol, Self, reveal_type, Literal

class MultiplyWithInt(Protocol):
    def __mul__(self, other: int, /) -> Self:
        ...

a: MultiplyWithInt = 1
b: MultiplyWithInt = 1.0
c: MultiplyWithInt = 1j
d: MultiplyWithInt = np.array([1], dtype=np.uint8) # type check error
e: MultiplyWithInt = np.array([1], dtype=np.uint16) # type check error
f: MultiplyWithInt = np.array([1], dtype=np.uint32) # type check error
g: MultiplyWithInt = np.array([1], dtype=np.uint64) # type check error
h: MultiplyWithInt = np.array([1], dtype=np.int8)
i: MultiplyWithInt = np.array([1], dtype=np.int16)
j: MultiplyWithInt = np.array([1], dtype=np.int32)
k: MultiplyWithInt = np.array([1], dtype=np.int64)
l: MultiplyWithInt = np.array([1], dtype=np.float32)
m: MultiplyWithInt = np.array([1], dtype=np.float64) # type check error
n: MultiplyWithInt = np.array([1], dtype=np.complex64)
o: MultiplyWithInt = np.array([1], dtype=np.complex128) # type check error

reveal_type(np.array([1], dtype=np.uint8)) # dtype[unsignedinteger[_8bit]]
reveal_type(np.array([1], dtype=np.uint8) * 1) # dtype[signedinteger[Any]]
reveal_type(np.array([1], dtype=np.uint8) * np.uint8(1)) # dtype[unsignedinteger[Any]]

reveal_type(np.array([1], dtype=np.int8)) # dtype[signedinteger[_8bit]]
reveal_type(np.array([1], dtype=np.int8) * 1) # dtype[signedinteger[Any]]
reveal_type(np.array([1], dtype=np.int8) * np.int8(1)) # dtype[signedinteger[Any]]

Should I open that as a new issue?

This should be fixed by #28108