my official "why can't there be nested generics" question

[zzzeek]

hey typing folks -

For SQLAlchemy, this is something I've been wrestling with for about a year now, and it looks like I've gotten it wrong in sqlalchemy2-stubs. But I want to put it out there to typing experts to help me understand what it is I'm looking at, to confirm it's definitely not possible, whether or not it's definitely never going to be possible either, or maybe it is, or could be, or not, or if I'm totally misunderstanding the whole issue it in which case I need the education :).

In https://www.python.org/dev/peps/pep-0484/#generics the concept of "generics" is discussed in terms of "type information about objects kept in containers". So here I will illustrate code that shows objects that "contain" another object of some type.

The issue is, I can't define a container of a container (call it "outer container" of "inner container" of "some object"), in such a way that "outer container" has static typing information about both the inner container and some object at the same time.

When I write that, I mentally have a picture of typing gurus queuing up the dreaded "it's too complicated and suggests your code is poorly architected to require that " response. which I probably don't agree with, but I don't yet see any "it's logically impossible" aspects of it.

First let's define the "inner container", a Python object with an instance variable of arbitrary type _T. There are also subclasses of "inner container" which are generic against specific subtypes of _T, and the type of these subclasses is also important, in that they may suggest other behaviors as well:

from typing import Generic, Any
from typing import TypeVar

_T = TypeVar("_T", bound=Any)


class InnerContainer(Generic[_T]):
    """an object that contains an arbitrary Python object of some type"""
    def __init__(self, item: _T):
        self.item = item


class StringInnerContainerOne(InnerContainer[str]):
    """Subclasses of InnerContainer exist which may define new behaviors."""
    def method_one(self) -> int:
        return 25

class StringInnerContainerTwo(InnerContainer[str]):
    """For example, there can be more than one subclass of InnerContainer that stores Python str objects."""

    def method_two(self) -> str:
        return "method two's return value"

Now we want to make a class OuterContainer that stores an object that's an InnerContainer, and we want to be able to type attributes / methods on OuterContainer both in terms of the type of InnerContainer as well as the contained type _T of InnerContainer. From everything I can see, this is not possible with current specifications.

Here's attempt one, which is how I've always viewed as the most elegant way to do this, but I now believe is wrong: make OuterContainer generic against the InnerContainer type. When we do this, OuterContainer can have all kinds of info about InnerContainer, but nothing that refers to its inner type:

_IC = TypeVar("_IC", bound="InnerContainer")


class OuterContainerOne(Generic[_IC]):
    """This container will be generic on the type of the InnerContainer."""

    def __init__(self, item: _IC):
        self.item = item

    def get_contained_thing(self) -> _IC:
        """we can type the contained thing.

        Here we would know if we had for example a StringInnerContainerOne
        or a StringInnerContainerTwo.

        """
        return self.item

    def get_thing_contained_by_thing(self) -> Any:
        """but we can't type the thing-contained-by-contained-thing.

        There's no syntax that can access the fact that InnerContainer
        has a type "_T".

        For example, we couldn't say -> _TE->_T or something like that, when we are 
        typed only against _TE and not _T, there's no syntax I'm aware of to 
        access this.

        So typing information is lost when we create OuterContainerOne.

        """
        return self.item.item

The next way, which is at the moment a lot more practical but still loses typing information, is to type the outer container against the type contained by inner container. This is the most workable system but loses typing information and also feels really wrong to me, because it makes it seem OuterContainer is a container for the innermost objects, and not for InnerContainer objects, which are lost here:

class OuterContainerTwo(Generic[_T]):
    """This container will be generic on the type of the thing
    contained by the InnerContainer."""

    def __init__(self, item: InnerContainer[_T]):
        """the constructor works great here and we can extract _T right out 
        from InnerContainer, by basically discarding anything else known 
        about the subtype of InnerContainer.

        """
        self.item = item

    def get_thing_contained_by_thing(self) -> _T:
        """now we can type the thing-contained-by-contained-thing.

        that's pretty good and useful.

        """
        return self.item.item

    def get_contained_thing(self) -> InnerContainer[_T]:
        """but for the immediate container, we can't get the exact type;
        we can for example see that it's InnerContainer[str] but we can't
        get information that it's about StringInnerContainerOne or
        StringInnerContainerTwo.

        So again, typing information is lost when we create OuterContainerTwo.

        """
        return self.item

the above example gets close, and for the SQLAlchemy problem we have, it looks like we're going to use that method. But when I have code like the following, typing information is lost:

oc = OuterContainerTwo(StringInnerContainerTwo("some string"))

# reveals as:  InnerContainer[str]
# and not: StringInnerContainerTwo[str]
reveal_type(oc.get_contained_thing)

for the finale of the question, here's the syntax that for me seems like it would solve these problems completely, while probably creating lots of new problems for the people that write the typing tools, but anyway, here it is. Would this syntax be something theoretically feasible someday? or is there some way I'm not seeing to achieve this result?

_IC = TypeVar("_IC", bound="InnerContainer")

class OuterContainerImpossible(Generic[_IC[_T]]):
    """the solution: create a Generic where I can type it against  
    the inner container type and then also get a handle on the innermost 
    contained type at once.

    this syntax is currently rejected.

    This syntax though would solve the problem, because we know _IC is
    InnerContainer, and we know what the subtype is, it's _T.

    """

    def __init__(self, item: _IC[_T]):
        self.item = item


    def get_contained_thing(self) -> _IC:
        """now we can type the contained-by thing.
        """
        return self.item

    def get_thing_contained_by_thing(self) -> _T:
        """and we can type the thing-contained-by-contained-thing.

        no typing information is lost; we can access everything about
        InnerContainer and what it contains.

        """
        return self.item.item

to reiterate, I know the above syntax is not supported. Looking here for the typing-centric explanation of whether or not it introduces unworkable problems, or otherwise why it's not worth doing. thanks for reading!

[sobolevn]

You might be looking for #548

We have a working prototype (that I cannot recommend for production) of HKTs in Python: https://returns.readthedocs.io/en/latest/pages/hkt.html

[zzzeek]

wow it exists!! and has a name! that's just what I was looking for!

Clearly we can't use it now :) but at least I can keep track of it if we want to do this someday.