Allow variadic generics #193
Comments
|
Hmm... why not make this a special property of type variables.Tthen you wouldn't need the funny syntax, you could just use a type variable that has a keyword saying it is variadic. But maybe the bigger question is how exactly we should type check this. |
This isn't obviously objectionable, but for the following reasons I'm not sure it's actually a good idea. First, I would like to point out that I failed to notice a third possible syntax, which doesn't require the trailing comma: I'll return to this syntax in a moment.
We should enforce that the variable only appears as the sole argument to Tuple (with a trailing ellipsis?), as a variable of a generic class, possibly (?) in the instantiation of a generic class other than Tuple, or as the type of a variadic parameter ( We can overcome both problems by requiring specialized syntax when the generic class is subclassed or instantiated: Note the extra pair of brackets. They indicate where the variadic typevar begins and ends, which removes any ambiguity when there are multiple variadic typevars, and simplifies parsing when there are non-variadic typevars in the same class (as in this case). For reasons of uniformity, I would recommend we use the bracket syntax I showed above rather than making variadic-ness a property of the type variable. That makes instantiation and declaration look more like one another, and seems more intuitive to me. |
|
Variadic generics would likely be useful at least occasionally. It would make sense to generalize them also to argument types in In my proposal a variadic type variable
It's less obvious how to generalize this to user-defined generic types. Maybe like this: The limitation would be that in order to have both variadic and non-variadic arguments, you'd have to create a dummy wrapper type for the variadic args: The apparent awkwardness of the above example probably wouldn't matter much as I expect this use case to be very rare. There are additional implementation details that a type checker should probably get right to make this useful, but I'm not even trying to enumerate them here. The implementation would likely be tricky, but probably not excessively so. I'm still unconvinced that this is a very important feature -- look at how long it took C++ to pick up this feature. I'd propose looking for more use cases to justify variadic generics and once we have a sufficient number of use cases collected here, we'd send this to python-ideas@ for discussion. |
|
Examples where these could be useful:
|
|
@JukkaL There's one extension to your proposal that would solve a problem I've been trying to figure out: I have some I am not at all wed to this syntax for mapping over variadic types variables, but I want to be able to do it, and I think it should be a thing we should consider when considering variadic type variables. |
|
It's not very clear from that example that execute() returns a tuple. I think Jukka's proposal would have you write Tuple[Rs] for the return type. I also wish the map() functionality was expressible as part of the signature of execute() rather than in the definition of Qs. Anyway, this would be shorthand for an infinite sequence of definitions for execute(), like this, right? R = TypeVar('R')
R1 = TypeVar('R1')
R2 = TypeVar('R2')
R3 = TypeVar('R3')
# etc. until R999
@overload
def execute(q1: Query[R]) -> R: ...
@overload
def execute(q1: Query[R1], q2: Query[R2]) -> Tuple[R1, R2]: ...
@overload
def execute(q1: Query[R1], q2: Query[R2], q3: Query[R3]) -> Tuple[R1, R2, R3]: ...
# etc. until a total of 999 variantsNOTE: The special case for the first overload is not part of the special semantics for variadic type variables; instead there could be two overload variants, one for a single query, one for 2 or more. |
|
Yes, |
|
And yes, this would be exactly that infinite series of overloads. |
|
Making the type mapping in the argument list would be nice, but requires some care as to exactly where you are parameterizing a type over each element of your variadic type variable, and where you are parameterizing a type using all elements of your variadic type variable. For example, I'd love to be able to write One possibility is, borrowing some stars and parens from @NYKevin to have, I think, somewhat of a different meaning: |
|
So maybe the notation ought to reflect that, and we should be able to write Rn = TypeVar('Rn', variadic=True)
def execute(*q1: Query[Rn.one]) -> Tuple[Rn.all]: ...Where .one and .all try to give hints on how to expand these. |
|
Ooh, I like not having it be some kind of obtuse operator but rather english words. Consider |
Unless, say, |
|
Here's "all is the default, you have to specify each" Here's "each is the default, you have to specify all" |
|
I need to think about more examples. But note that we're talking about
*queries...
…--Guido (mobile)
|
|
For another example, here's As = TypeVar('As', variadic=True)
R = TypeVar('R')
def map(f: Callable[[As.all], R], *args: Iterable[As]) -> Iterable[R]
...And here's using "all is the default" form: As = TypeVar('As', variadic=True)
R = TypeVar('R')
def map(f: Callable[[As], R], *args: Iterable[As.each]) -> Iterable[R]
... |
|
I think that in each location, only one of Here's the Note that |
|
@JukkaL It might be a contrived example, but what about this (written with super-explicit each/all notation, to be clear about where you might find ambiguity): Ts = TypeVar('Ts', variadic=True)
def make_lots_of_unary_tuples(*scalars: Ts) -> Tuple[Tuple[Ts.each].all]:
return tuple(tuple([s]) for s in scalars) |
|
Vs this: Ts = TypeVar('Ts', variadic=True)
def make_a_tuple_wrapped_in_a_tuple(*scalars: Ts) -> Tuple[Tuple[Ts.all]]:
return tuple([tuple(scalars)]) |
|
The types aren't quite right in your examples, since |
|
So the rule is that parameterizing something by a variadic type either results in another variadic type which represents one instance of the parametrized type per captured type variable in the variadic ( Yeah, I'll tweak the examples. |
|
Yeah, those look reasonable, though pretty contrived :-). I'm still not convinced that Another thing to consider is type checking code that uses variadic type variables. Supporting them in stubs only could be much easier than supporting them in function bodies. |
|
Another exploration of the design space of syntax: |
|
Ok, here I am a couple days later with a partially-working implementation under my belt. Here's what I think will work (I'm reëxplaining a decent amount here, and it's a wall of text, but bear with me): SyntaxYou create a variadic type variable with @JukkaL 's syntax: Variadic type variables are contagious; when you use one as a type argument to some other type, it too becomes variadic: To expand a variadic type in function arguments, give the variadic type as the type of a Without a return type, that particular type signature is not particularly useful. Aside from
ExamplesWith this syntax, here are the discussed use-case signatures (I think): Ts = TypeVar('Ts', variadic=True)
R = TypeVar('R')
def zip(*args: Iterable[Ts]) -> List[Tuple[Ts, ...]]: ...
def map(fn: Callable[[Ts, ...], R], *args: Iterable[Ts]) -> List[R]: ...
def execute_all(*args: Query[Ts]) -> Tuple[Ts, ...]: ...Some functions take some positional fixed arguments and some variadic arguments. For example, here's a version of def map_with_idx(fn: Callable[[int, Ts, ...], R], *args: Iterable[Ts]) -> List[R]:
results = [] # type: List[R]
for i, rotated_args in enumerate(zip(*args)):
results.append(fn(i, *rotated_args))
return results(By extension, Details: Function body typecheckingTo typecheck the function body, there is very little we can conclude about the expansion of a variadic type -- the implementation of nearly any non-trivial function will almost certainly iterate over its # This is how the typechecker sees it when checking the body
def map_with_idx(fn: Callable[[int, Any, ...], R], *args: Iterable[Any]) -> List[R]:
results = [] # type: List[R]
for i, rotated_args in enumerate(zip(*args)):
results.append(fn(i, *rotated_args))
return resultsDetails: Splatting arguments inIn the implementation of The true meaning of
Details: Implementation!!!I implemented part of what I described. I might actually get to a little more tomorrow. What I've got so far:
The buried lede: |
|
The link to the implementation above should now support full typechecking at the callsite of the following functions, as long as you don't splat args into them (and a fallback to some kind of reasonable use of Ts = TypeVar('Ts', variadic=True)
R = TypeVar('R')
def my_zip(*args: Iterable[Ts]) -> Iterator[Tuple[Ts, ...]]:
iterators = [iter(arg) for arg in args]
while True:
yield tuple(next(it) for it in iterators)
def make_check(*args: Ts) -> Callable[[Ts, ...], bool]:
"""Return a function to check whether its arguments are the same as this function's args"""
def ret(*args2: Ts) -> bool:
if len(args) != len(args2):
return False
for a, b in zip(args, args2):
if a != b:
return False
return True
return ret
def my_map(f: Callable[[Ts, ...], R], *args: Iterable[Ts]) -> Iterator[R]:
for parameters in zip(*args):
yield f(*parameters) |
|
(It's not pull-request-ready by any means, but all the type system features are there, just not all the error messages and not-crashing-if-you-do-something-I-didn't-think-of) |
|
FWIW, just to play devil's advocate, here's a more pedestrian approach: https://gist.github.com/gvanrossum/86beaced733b7dbf2d034e56edb8d37e |
|
Hi all. Thank you for mypy! So, what is the plan going forward regarding variadic generics? |
The previous link was to "extended callable" types, which are
deprecated in favor of callback protocols. Unfortunately, defining a
protocol class can't express the typing -- we need some sort of
variadic generics[1]. Specifically, we wish to support hitting the
endpoint with additional parameters; thus, this protocol is
insufficient:
```
class WebhookHandler(Protocol):
def __call__(request: HttpRequest, api_key: str) -> HttpResponse: ...
```
...since it prohibits additional parameters. And allowing extra
arguments:
```
class WebhookHandler(Protocol):
def __call__(request: HttpRequest, api_key: str,
*args: object, **kwargs: object) -> HttpResponse: ...
```
...is similarly problematic, since the view handlers do not support
_arbitrary_ keyword arguments.
[1] python/typing#193
The previous link was to "extended callable" types, which are
deprecated in favor of callback protocols. Unfortunately, defining a
protocol class can't express the typing -- we need some sort of
variadic generics[1]. Specifically, we wish to support hitting the
endpoint with additional parameters; thus, this protocol is
insufficient:
```
class WebhookHandler(Protocol):
def __call__(request: HttpRequest, api_key: str) -> HttpResponse: ...
```
...since it prohibits additional parameters. And allowing extra
arguments:
```
class WebhookHandler(Protocol):
def __call__(request: HttpRequest, api_key: str,
*args: object, **kwargs: object) -> HttpResponse: ...
```
...is similarly problematic, since the view handlers do not support
_arbitrary_ keyword arguments.
[1] python/typing#193
The previous link was to "extended callable" types, which are
deprecated in favor of callback protocols. Unfortunately, defining a
protocol class can't express the typing -- we need some sort of
variadic generics[1]. Specifically, we wish to support hitting the
endpoint with additional parameters; thus, this protocol is
insufficient:
```
class WebhookHandler(Protocol):
def __call__(request: HttpRequest, api_key: str) -> HttpResponse: ...
```
...since it prohibits additional parameters. And allowing extra
arguments:
```
class WebhookHandler(Protocol):
def __call__(request: HttpRequest, api_key: str,
*args: object, **kwargs: object) -> HttpResponse: ...
```
...is similarly problematic, since the view handlers do not support
_arbitrary_ keyword arguments.
[1] python/typing#193
The previous link was to "extended callable" types, which are
deprecated in favor of callback protocols. Unfortunately, defining a
protocol class can't express the typing -- we need some sort of
variadic generics[1]. Specifically, we wish to support hitting the
endpoint with additional parameters; thus, this protocol is
insufficient:
```
class WebhookHandler(Protocol):
def __call__(request: HttpRequest, api_key: str) -> HttpResponse: ...
```
...since it prohibits additional parameters. And allowing extra
arguments:
```
class WebhookHandler(Protocol):
def __call__(request: HttpRequest, api_key: str,
*args: object, **kwargs: object) -> HttpResponse: ...
```
...is similarly problematic, since the view handlers do not support
_arbitrary_ keyword arguments.
[1] python/typing#193
reason being it's currently impossible to return a correct type with mypy python/typing#193
reason being it's currently impossible to return a correct type with mypy python/typing#193
|
A few of us have actually been working on a draft of a PEP for variadics generics since last year, PEP 646. Eric Traut has very kindly contributed an initial implementation in Pyright, Pradeep Kumar Srinivasan has been working on an implementation in Pyre, and we're working on the additions to @NYKevin As of the current draft, I think your use case with GetSetVar = TypeVar('GetSetVar')
TagVar = TypeVarTuple('TagVar')
class MultiField(AbstractField[GetSetVar], Generic[*TagVar]):
def __init__(self, nbt_names: Sequence[str], *, default: GetSetVar = None) -> None:
...
@abc.abstractmethod
def to_python(self, *tags: *TagVar) -> GetSetVar:
...
@abc.abstractmethod
def from_python(self, value: GetSetVar) -> Tuple[*TagVar]:
...@sixolet I think you've seen the thread in typing-sig about 646, but for the sake of other people reading this thread: the R = TypeVar('R')
Rs = TypeVarTuple('Rs')
class Query(Generic[R]):
...
def execute(*args: *Map[Query, Rs]) -> Tuple[*Rs]:
...@seaders I think your use case should be possible with: Ts = TypeVarTuple('Ts')
@classmethod
def qwith_entities(cls, models: Tuple[*Ts]) -> List[Tuple[*Ts]]:
return cls.query.with_entities(*models)In any case, if you have any feedback on the current draft of the PEP, please do drop by the thread in typing-sig and leave us a message. Cheers! |
srittau
added
topic: feature
|
Does I'm trying to annotate the following function, but I'm not sure this is supported. It's unclear from the PEP as there's no example. _KeyT = TypeVar('_KeyT')
_ValuesT = TypeVarTuple('_ValuesT')
def zip_dict(*dicts: *dict[_KeyT, _ValuesT]) -> Iterable[_KeyT, tuple[*_ValuesT]]:
"""Iterate over items of dictionaries grouped by their keys."""
for key in set(itertools.chain(*dicts)):
yield key, tuple(d[key] for d in dicts) |
|
@Conchylicultor this is unsupported. There's been talk of adding a Closing this issue as PEP 646 has been accepted and implemented (thanks @mrahtz and @pradeep90!). |
C++11 recently introduced the notion of variadic templates, which I believe Python could benefit from in a simplified form.
The idea is that you can have a generic class with a variable number of type variables, like
typing.Tuple[]has. Here is a real-world variadic-generic class which is notTuple; it is variadic inTagVar. As you can see,TagVaronly appears in tuple contexts. Those tuples are sometimes heterogenous (Caution: annotations are a homegrown 3.4-compatible mishmash of nonsense), so repeatingTagVaras shown is actually incorrect (but the closest approximation I could find).Here's one possible syntax:
This is syntactically valid in Python 3.5 (if a bit ugly with the parentheses and trailing comma, which cannot be omitted without language changes), but doesn't currently work because type variables are not sequences and cannot be unpacked. It could be implemented by adding something like this to the TypeVar class:
StarredTypeVar would be a wrapper class that prefixes the repr with a star and delegates all other functionality to the wrapped TypeVar.
Of course, syntax isn't everything; I'd be fine with any syntax that lets me do this. The other immediately obvious syntax is to follow the TypeVar with an ellipsis, which conveniently does not require changes to typing.py. However, that might require disambiguation in some contexts (particularly since
Tupleis likely to be involved with these classes).The text was updated successfully, but these errors were encountered: