maximum arity of itertools.product() is hardcoded to 10. document, increase, or print a warning?

[muxator]

Hi,

for reasons that I can't control, I have to support a software that calls itertools.product() with an arity greater than 10. I use mypy (currently 1.15.0) and python 3.13.1, but the problem is largely independent from the software versions.

It took me a couple of days to realize that this case is not supported, because the maximum arity for itertools.product() is controlled by 10 manually written overloads, one for each arity.
For example, the overload for product()/10 is here:

typeshed/stdlib/itertools.pyi

Lines 261 to 274 in 24c78b9

	def __new__(
	cls,
	iter1: Iterable[_T1],
	iter2: Iterable[_T2],
	iter3: Iterable[_T3],
	iter4: Iterable[_T4],
	iter5: Iterable[_T5],
	iter6: Iterable[_T6],
	iter7: Iterable[_T7],
	iter8: Iterable[_T8],
	iter9: Iterable[_T9],
	iter10: Iterable[_T10],
	/,
	) -> product[tuple[_T1, _T2, _T3, _T4, _T5, _T6, _T7, _T8, _T9, _T10]]: ...

The problem I faced is that the type checker (mypy 1.14.x and 1.15.x in my case), instead of clearly asserting that it could not work with arities greater than 10, spit out wrong deductions (see the example later on).

The obvious desiderata here would be not having to rely on any hardcoded overloads and support functions of arbitrary arity. I suppose this requires a more powerful type system, or checker, or both. In each case I do not expect it to happen in a short time, right?

However, it took me a couple of days to realize the existence of this limitation in the type deduction of itertools.product(), because I kept trying to find a bug in the code I have to maintain; I'd like to spare this suffering to future users.

Do you think at least one of the following is possible?

1. find a way to cause a warning in the type checkers, so that users know that they are experiencing a limitation in the stubs/type checker and not in their code
2. increment the number of supported overloads for itertools.product() from 10 to 15, so that the limitation is hit more rarely. I have manually altered itertools.pyi in my venv, and it makes mypy happy (and cause misery to the idealists among us)
3. document the limitation on the maximum arity of itertools.product(), for example in https://typing.readthedocs.io/

If 2. is acceptable, I am willing to submit a PR. With enough hand holding, I could also contribute point 3, if it helps.

Thank you very much.

Example of a program that breaks badly when iterable_11 is uncommented:

from itertools import product
from typing import Literal


def f(
    iterable_01: tuple[Literal["one"]] | list[int],
    iterable_02: list[Literal[2]],
    iterable_03: list[Literal[3]],
    iterable_04: list[Literal[4]],
    iterable_05: list[Literal[5]],
    iterable_06: list[Literal[6]],
    iterable_07: list[Literal[7]],
    iterable_08: list[Literal[8]],
    iterable_09: list[Literal[9]],
    iterable_10: list[Literal[10]] | tuple[None, Literal[10]],
    # iterable_11: list[Literal[11]],
) -> None:
    for (
        elem_01,
        elem_02,
        elem_03,
        elem_04,
        elem_05,
        elem_06,
        elem_07,
        elem_08,
        elem_09,
        elem_10,
        # elem_11,
    ) in product(
        iterable_01,
        iterable_02,
        iterable_03,
        iterable_04,
        iterable_05,
        iterable_06,
        iterable_07,
        iterable_08,
        iterable_09,
        iterable_10,
        # iterable_11,
    ):
        reveal_type(elem_01)
        reveal_type(elem_02)
        reveal_type(elem_03)
        reveal_type(elem_04)
        reveal_type(elem_05)
        reveal_type(elem_06)
        reveal_type(elem_07)
        reveal_type(elem_08)
        reveal_type(elem_09)
        reveal_type(elem_10)

Type checking with 10 arguments and correct type deduction:

$ mypy itertools_product.py 
itertools_product.py:43: note: Revealed type is "Literal['one']"
itertools_product.py:44: note: Revealed type is "Literal[2]"
itertools_product.py:45: note: Revealed type is "Literal[3]"
itertools_product.py:46: note: Revealed type is "Literal[4]"
itertools_product.py:47: note: Revealed type is "Literal[5]"
itertools_product.py:48: note: Revealed type is "Literal[6]"
itertools_product.py:49: note: Revealed type is "Literal[7]"
itertools_product.py:50: note: Revealed type is "Literal[8]"
itertools_product.py:51: note: Revealed type is "Literal[9]"
itertools_product.py:52: note: Revealed type is "Union[Literal[10], None]"
Success: no issues found in 1 source file

Type checkng with 11 arguments and incorrect type deduction (this is random: sometimes mypy writes builtins.object, instead):

$ mypy itertools_product.py 
itertools_product.py:43: note: Revealed type is "Union[Literal['one'], builtins.int, None]" <-- wider, but wrong
itertools_product.py:44: note: Revealed type is "Union[Literal['one'], builtins.int, None]" <-- completely wrong
itertools_product.py:45: note: Revealed type is "Union[Literal['one'], builtins.int, None]" [...]
itertools_product.py:46: note: Revealed type is "Union[Literal['one'], builtins.int, None]" [...]
itertools_product.py:47: note: Revealed type is "Union[Literal['one'], builtins.int, None]" [...]
itertools_product.py:48: note: Revealed type is "Union[Literal['one'], builtins.int, None]" [...]
itertools_product.py:49: note: Revealed type is "Union[Literal['one'], builtins.int, None]" [...]
itertools_product.py:50: note: Revealed type is "Union[Literal['one'], builtins.int, None]" [...]
itertools_product.py:51: note: Revealed type is "Union[Literal['one'], builtins.int, None]" [...]
itertools_product.py:52: note: Revealed type is "Union[Literal['one'], builtins.int, None]" <-- completely wrong
Success: no issues found in 1 source file

[TeamSpen210]

Maybe it’d be a good idea to @deprecate the final fallback overload in all these manually defined functions, to indicate that types will be inaccurate.

[hauntsaninja]

I'd be strongly opposed to use of deprecated here.

Note the type isn't completely wrong, just too broad. I'd potentially be open to some Any based solution though.

In general, 10 is a less surprising choice of cliff if one must have a surprising cliff.

[muxator]

Note the type isn't completely wrong, just too broad

Yep, in this case the thing that comes out is something understandable.

My actual code was more complex, and mypy computed a monster-type 15 lines long. Additionally, it did not give stable results across different executions.

While there were 10 parameters, the code and its deducted types were sound. As soon as I added another one, I started seeing typing errors, but no mention of a problem on mypy's side.

I think that I'd be disoriented all the same if mypy started classifying types as "Any" or "object" as soon as I added a parameter, without telling me that the behavior is due to a limitation in the typing system, instead of a problem in my code.

Is there a way to generate this kind of warning?

[muxator]

About the hard limit to an arity of 10 for product(): this is certainly more than enough for code of reasonable good quality.

However, code bases of bad quality are extremely frequent. Those are the ones that might benefit the most from static type checking.

For this reason I am starting to think that, as dumb as it sounds, it might be a good idea to raise the limit to 15 or 20, as well as finding a way to make this limit more evident.

Does it make sense? How does it sit with the maintainers?

[Akuli]

I think raising the limit to 20 is a good idea. For hard limits like this, my general rule of thumb tends to be "biggest anyone might reasonably need times two", which is 10*2 here.

Edit: Raising the limit might make "monster types" and error messages even worse, maybe 15 as you suggested?

[AlexWaygood]

I'm fine with raising the limit if there's real user code that would benefit from it (here there clearly is, or the issue wouldn't have been opened!). But I also think we shouldn't raise the limit more than is actually necessary. As well as "monster types" and unreadably long error messages, having too many overloads also comes with the cost that it can make type checkers less performant when analysing code that uses itertools.product().

[JelleZijlstra]

Additionally, it did not give stable results across different executions.

That seems like a mypy bug, so you should report it to mypy.

[muxator]

@Akuli:

I think raising the limit to 20 is a good idea. For hard limits like this, my general rule of thumb tends to be "biggest anyone might reasonably need times two", which is 10*2 here.

Edit: Raising the limit might make "monster types" and error messages even worse, maybe 15 as you suggested?

I agree, let's be frugal: I've submitted a PR that increases the limit to 15 instead of 20.

I'd like to point out that in my case the "monster error message" was caused by the lack of overloads. Once I manually patched my local typeshed the program passed every check and everything was very smooth. 😄

[muxator]

@AlexWaygood:

I'm fine with raising the limit if there's real user code that would benefit from it (here there clearly is, or the issue wouldn't have been opened!). But I also think we shouldn't raise the limit more than is actually necessary. [...], having too many overloads also comes with the cost that it can make type checkers less performant when analysing code that uses itertools.product().

Yep. I went with 15 in order to avoid this problem. I know no easy way of measuring the performance impact of these kind of changes. I guess it depends on how each type checker is implemented (a topic that is out of my specialization).

I'd imagine that, given a product()/n invocation, a type checker would do a direct hashtable lookup on an internal structure, and that we just incremented that table size from 10 to 15 entries. But I do not want to speculate too much.

On a more serious plane, what interested me is the necessity of manually specifying a given number of overloads. It seems that the type hinting language lacks a way of expressing this property in a generic way. Or maybe this is really a problem with no theoretical solution. Just curious.

[muxator]

@JelleZijlstra

That seems like a mypy bug, so you should report it to mypy.

You are right. I'll try to find a minimal reproducer and submit it to mypy, thanks.

[muxator]

Just for the sake of documentation, here are two discussions that are related to properly solving this problem:

• Map over types in a type-checker-friendly way typing#1450
• TypeVarTuple Transformations Before Unpack typing#1216

Probably for the same reason, the type annotations in map() are structured in the same way as those in product() (they use explicit overloads).