does pep-646 let me make a MyTuple[_T, ...] of Element[_T] or am i misunderstanding

[zzzeek]

Starting with a really basic generic class:

from typing import Generic, TypeVar, Tuple

_T = TypeVar("_T")

class Element(Generic[_T]):
    def __init__(self, item: _T):
        self.item = item

# e1: Element[str]
e1 = Element("a string")

# e2: Element[int]
e2 = Element(45)

we can make a new class that is given an Element as a constructor argument, and is then itself a class that is also Generic[_T]:

class ContainerOfElement(Generic[_T]):
    def __init__(self, element: Element[_T]):
        self.element = element

#  c1: ContainerOfElement[str]
c1 = ContainerOfElement(e1)

my hope for pep-646, which I'm not sure now is covered at all, is the exact same pattern as above, except for a ContainerOfMultipleElements. This seems like the most simple, basic thing that would be allowed from a typing extension that grants the ability to make custom tuple-like classes, and for all the ORM use cases that I've seen discussed in some older pep-646 discussions, this would be the bare minimum thing needed. I'm currently working with the pep-646 support in pyright to test, but with no luck. Is the below possible? that is, a MyTuple[_T, ...] that's constructed from a Tuple[Element[_T], ...] ?:

from typing_extensions import TypeVarTuple
from typing_extensions import Unpack

MyTuple = TypeVarTuple("MyTuple")


class ContainerOfMultipleElements(Generic[Unpack[MyTuple]]):
    def __init__(self, elements: Tuple[Unpack[MyTuple]]):
        """doesn't work, gives me Element[str] / Element[int] as datatype"""

        self.elements = elements

    # def __init__(self, elements: Tuple[Unpack[MyTuple[_T] ??? ]]):
    #     """Need a way to move from MyTuple to _T"""
    #     self.elements = elements


# desired:
# t1: ContainerOfMultipleElements[str, int]

# what we actually get:
# t1: ContainerOfMultipleElements[Element[str], Element[int]]

t1 = ContainerOfMultipleElements((e1, e2))

[erictraut]

I think what you're looking for is a Map operation over the TypeVarTuple contents. There was something like this in an earlier version of PEP 646, but it was removed because the PEP was already very complex. It's something that could be added in a follow-on PEP assuming that 646 is accepted.

[zzzeek]

OK! well that pretty much ends pep-646 being meaningful for SQLAlchemy for the moment, thanks for the reply!

[zzzeek]

Hi all -

Here's a hardcoded way to make my above case work. pyright understands the end result and comes up with the right answer. but it still emits an error that what I'm doing is illegal. @erictraut is there a way to make this work? Shouldn't a pep-646 tuple, when passed around in "unpack" form, be able to be represented in a way that's agnostic of how many elements it has?

from typing import Any
from typing import Generic
from typing import Optional
from typing import overload
from typing import TypeVar

from typing_extensions import TypeVarTuple
from typing_extensions import Unpack

_T = TypeVar("_T")

_T1 = TypeVar("_T1")
_T2 = TypeVar("_T2")
_T3 = TypeVar("_T3")
_T4 = TypeVar("_T4")


class Element(Generic[_T]):
    def __init__(self, item: _T):
        self.item = item


# e1: Element[str]
e1 = Element("a string")

# e2: Element[int]
e2 = Element(45)

e3 = Element("str")


MyTuple = TypeVarTuple("MyTuple")


class ContainerOfMultipleElements(Generic[Unpack[MyTuple]]):
    def __init__(self, *elements: Element[Any]):
        ...



@overload
def container(
    e1: Element[_T1], e2: Element[_T2], e3: Element[_T3]
) -> ContainerOfMultipleElements[_T1, _T2, _T3]:
    ...

@overload
def container(
    e1: Element[_T1], e2: Element[_T2]
) -> ContainerOfMultipleElements[_T1, _T2]:
    ...


@overload
def container(
    e1: Element[_T1],
) -> ContainerOfMultipleElements[_T1]:
    ...


@overload
def container(
    e1: Optional[Element[Any]] = None,
    e2: Optional[Element[Any]] = None,
    e3: Optional[Element[Any]] = None,
    *elements: Element[Any],
) -> ContainerOfMultipleElements[Unpack[MyTuple]]:
    ...


# pylance understands the types, but complains:
#
# Function return type "ContainerOfMultipleElements[_T1@container, _T2@container, _T3@container]" is incompatible with type "ContainerOfMultipleElements[*MyTuple@container]"
#      TypeVar "MyTuple@ContainerOfMultipleElements" is invariant
#        Element size mismatch; expected 0 but received 3

def container(
    e1: Optional[Element[Any]] = None,
    e2: Optional[Element[Any]] = None,
    e3: Optional[Element[Any]] = None,
    *elements: Element[Any],
) -> ContainerOfMultipleElements[Unpack[MyTuple]]:
    ...

# pylance sees these without issue:

# (variable) t1: ContainerOfMultipleElements[str]
t1 = container(e1)

# (variable) t2: ContainerOfMultipleElements[str, int]
t2 = container(e1, e2)

# (variable) t3: ContainerOfMultipleElements[str, int, str]
t3 = container(e1, e2, e3)

[erictraut]

A TypeVarTuple is similar to a TypeVar in that it is meant to be "solved" — i.e. bound to an actual type — when a call is evaluated. If you include a TypeVar in the return type of a function, the input parameters of the function need to reference that same TypeVar. Otherwise there's no way for the constraint solver to determine the solution.

# This generic function signature doesn't make sense because there's 
# no way for the constraint solver to solve for T.
def func1(a: str, b: int) -> List[T]: ...

Likewise, in your example above, you're attempting to use a TypeVarTuple called MyTuple in a function return type, but it is not referenced by the input parameters, so there's no way for the constraint solver to solve it.

def container(
    e1: Optional[Element[Any]] = None,
    e2: Optional[Element[Any]] = None,
    e3: Optional[Element[Any]] = None,
    *elements: Element[Any],
) -> ContainerOfMultipleElements[Unpack[MyTuple]]:
    ...

Since the return type cannot be solved, the resulting type is effectively ContainerOfMuiltipleElements[Tuple[Any, ...]]. In this case, you should simply use Tuple[Any, ...] as you have done in your working example.

You are correct that PEP 646 doesn't help in this case. You would need a Map operation, which doesn't exist in the final version of PEP 646.

I'm having a hard time seeing what pep-646 is useful for.

The PEP describes the primary use cases that TypeVarTuple was designed to handle. Its main goal was to allow for the typing of tensors, which can have an arbitrary number of dimensions. That's obviously different from your use case.

[zzzeek]

So I am interested in understanding what the PEP process would be such that the notion of being able to create user-defined classes that behave like Tuple objects would be possible. This would include things like __getitem__() being able to work with integer indexes, being able to transfer between different sub-tuple objets like a Select[int, str, bool] given to a function can be returned as a Result[int, str, bool], things like that.

As it stands, the Tuple restriction is overly rigid. SQLAlchemy has the concept of a Result object that iterates Row objects. The Row behaves just like a Tuple. But I can't subclass Tuple, nor can I have Row.__getitem__() be typed against a contained Tuple.

I seem to recall the original discussions for pep-646 seemed to talk a lot about ORMs, but it looks like at the end of the day, ORM use cases aren't part of this pep at all. So what would the starting point be for a PEP that is about the use cases that ORMs and other similar tools that deal with arbitrary tuples be ?

[zzzeek]

specifically, I mean would pep-646 be the basis for such a pep, or is what Im describing again within the "non-goals" of pep-646 and there would be some totally different TupleLikeGeneric object introduced? it certainly seems like TypeVarTuple would be the basis for it but again I don't want to run over that "non-goal" blocker.

[erictraut]

PEP 646 has been accepted and is complete, so its goals and non-goals wouldn't apply to new PEPs. If you want to create a new PEP that builds upon the foundation that PEP 646 has established, you're welcome to do so. The authors of PEP 646 undoubtedly have some thoughts about future extensions, so it's probably best to work with them. I recommend reaching out to Matthew Rahz (his contact information is in the PEP) to initiate a discussion. Matthew hosted monthly get-togethers for the better part of a year to help PEP 646 coalesce into its final form.

I'll note that PEP 646 in its current form hasn't yet been implemented in mypy. Until that support is added, it will be difficult for library authors to use PEP 646 features, much less new features that build upon TypeVarTuple as a foundation.

[zzzeek]

for sure, and I am designing workarounds based on Tuple for now not unlike what other tools are doing. I'm also aware that the vast majority of peps have a long discussion process so I know that pep-646 went through such a process, but there was not an option for me to be involved with that at the time as SQLAlchemy's development cycle was in no shape at all for typing two years ago nor had I had any experience doing large scale typing projects, so I would not have had reasonable input at that time.