Cannot use non-string annotations for `argparse._SubParsersAction` parameters

[OddBloke]

In #6178, argparse._SubParsersAction was modified to be generic. This poses a problem when using this type to annotate a function, because argparse._SubParsersAction is not subscriptable but, being generic, it won't pass strict type-checking without subscripting. Consider this example file:

from typing import Any
import argparse


def example(subparser: argparse._SubParsersAction) -> None:
    subparser.add_parser("subexample")

Python will execute this code, but mypy --strict reports: example.py:5: error: Missing type parameters for generic type "_SubParsersAction".

If we add a type parameter to satisfy mypy:

from typing import Any
import argparse


def example(subparser: argparse._SubParsersAction[Any]) -> None:
    subparser.add_parser("subexample")

then mypy --strict is happy, but Python cannot execute the file because TypeError: 'type' object is not subscriptable.

This can be worked around with a string annotation (i.e. "argparse._SubParsersAction[Any]") but this can't be assigned to a type variable, which makes all of my helper function definitions much noisier and multi-lined.

(My usecase: I have common subcommands that I use helper functions to add: these all accept a _SubParsersAction.)

[AlexWaygood]

When you say:

This can be worked around with a string annotation (i.e. "argparse._SubParsersAction[Any]") but this can't be assigned to a type variable, which makes all of my helper function definitions much noisier and multi-lined.

What exactly do you mean by "this can't be assigned to a type variable"? Can you give a concrete example of the thing you'd like to do but can't do? :)

[OddBloke]

When you say:

This can be worked around with a string annotation (i.e. "argparse._SubParsersAction[Any]") but this can't be assigned to a type variable, which makes all of my helper function definitions much noisier and multi-lined.

What exactly do you mean by "this can't be assigned to a type variable"? Can you give a concrete example of the thing you'd like to do but can't do? :)

Thanks for the speedy reply! I would like to be able to do:

from typing import Any
import argparse

_SubparserType = argparse._SubParsersAction[Any]


def example(subparser: _SubparserType) -> None:
    subparser.add_parser("subexample")

but this exhibits the same issues as the equivalent example in the report. (And when I say the workaround doesn't work, I mean that wrapping the subscripting in a string then doesn't work: Variable "example._SubparserType" is not valid as a type. I wouldn't expect this to work, to be clear.)

(N.B. I'm on Python 3.8, so I don't have access to typing.TypeAlias; I believe that using that with a stringified variable assignment would allow me to assign the work-around to a variable.)

[AlexWaygood]

(N.B. I'm on Python 3.8, so I don't have access to typing.TypeAlias; I believe that using that with a stringified variable assignment would allow me to assign the work-around to a variable.)

Have you tried using typing_extensions.TypeAlias? (typing_extensions is a third-party package, but it's one of mypy's dependencies, so if you're using mypy, you likely already have it installed, even if you don't realise it :)

[OddBloke]

Have you tried using typing_extensions.TypeAlias? (typing_extensions is a third-party package, but it's one of mypy's dependencies, so if you're using mypy, you likely already have it installed, even if you don't realise it :)

This did work (thanks!), and in attempting to avoid introducing a runtime dependency on typing_extensions I realised there's an even simpler workaround:

from typing import Any, TYPE_CHECKING
import argparse

if TYPE_CHECKING:
    _SubparserType = argparse._SubParsersAction[argparse.ArgumentParser]
else:
    _SubparserType = Any


def example(subparser: _SubparserType) -> None:
    subparser.add_parser("subexample")

Thank you for helping me get to this!

[AlexWaygood]

Thank you for helping me get to this!

No worries! Are you happy for this issue to be closed now?

[OddBloke]

No worries! Are you happy for this issue to be closed now?

_SubParsersAction is not subscriptable even in 3.11-rc, and it does feel like I should be able to specify this using valid Python code, rather than having to use invalid code inside a TYPE_CHECKING guard or string annotation (whether in the function signature or as an explicit TypeAlias). My contract is that I want to be passed a generic _SubParsersAction, which seems like the most common way to want it to be passed as a parameter, and we can't currently express that using valid Python code (due, somewhat ironically, to the class being made generic 😁).

It would be nice to avoid this pain for other users in future, but I'm unblocked (and don't have the time to get into the nitty-gritty of stdlib generics, unfortunately), so I'll leave it for you to decide.

[AlexWaygood]

rather than having to use invalid code inside a TYPE_CHECKING guard or string annotation

Well, "invalid code" feels slightly subjective :)

"Classes which are generic in stubs but not at runtime" is sort of an unsolved problem, to be honest. The solution implicit in PEP 585 is "just open a BPO issue asking for __class_getitem__ to be added to the class". But that doesn't really scale, because typeshed development happens at a faster pace than CPython development, and we're often making more and more classes generic so that type checkers can have more precise type inference.

Note that in general, we're more trigger-happy in making undocumented, private classes, like argparse._SubParsersAction, generic, since end users probably shouldn't be accessing those directly anyway 😉

If you don't like the if TYPE_CHECKING solution, by the way, here's an "evil hack" that you could also use:

import argparse
from types import GenericAlias

setattr(argparse._SubParsersAction, "__class_getitem__", classmethod(GenericAlias))
_SubparserType = argparse._SubParsersAction[argparse.ArgumentParser]  # hey look, _SubParsersAction is now generic!

[OddBloke]

rather than having to use invalid code inside a TYPE_CHECKING guard or string annotation

Well, "invalid code" feels slightly subjective :)

Apologies, I didn't mean that to come across as a judgement! It's code which no Python interpreter can successfully execute at runtime, that's all I meant.

"Classes which are generic in stubs but not at runtime" is sort of an unsolved problem, to be honest.

Yeah, I think this is ultimately the problem. It's not unreasonable to make this class generic: it solves a real problem for other typing users, but it has other knock-on issues like this one. Do we have an open issue for this broader issue already?

One potential solution that occurs to me would be to have separate genericised stubs which users can opt into through installation, and the ones which ship with mypy by default match the stdlib exactly (at the cost of the inference improvements you describe). Then people who opt into the alternate stubs can carry whatever code (or "evil hacks" 😁) they need to realign the alternate stubs with their usage of the stdlib, and people using mypy will have stubs that are as closely aligned with the stdlib as possible.

Note that in general, we're more trigger-happy in making undocumented, private classes, like argparse._SubParsersAction, generic, since end users probably shouldn't be accessing those directly anyway 😉

Yeah, that's a reasonable position to take, for sure. In this case though, when an instance of a class is returned by the public API of a module then I don't know if it should be fully considered private? My code isn't accessing anything namespaced as private directly here, all I'm doing is passing the return value of ArgumentParser.add_subparsers into a typed function. It's actually only typing that requires me to treat it as anything other than an undocumented, private class, if you know what I mean. I'd honestly prefer to not specify it (it is private, after all), but if I want type checking in functions which use it, I have no choice. (Of course, there are many very good reasons why it works this way, I'm not suggesting we should change how inference works, but I do think it means typeshed's "public" interface is broader than if these classes could be treated as completely anonymous.)

[AlexWaygood]

Do we have an open issue for this broader issue already?

I can't recall one off the top of my head, but it's come up frequently, and I think the problem is growing. Since this issue affects the whole typing community, rather than just typeshed, discussion should happen in the https://github.com/python/typing repo or the typing-sig mailing list — feel free to start a discussion in either of those places!

In this case though, when an instance of a class is returned by the public API of a module then I don't know if it should be fully considered private?

Yeah, this is also a tricky one — there's a similar issue with contextlib._GeneratorContextManager, which is nominally private and undocumented, but is sort of de facto public API since it's the return type of contextlib.contextmanager.

I don't think there's anything immediately actionable for typeshed here, so I'm closing this for now — but I completely agree with you that the current situation is imperfect on several levels :/ sorry I don't have any better solutions

[hauntsaninja]

In case it's useful as reference, mypy has documentation for this class of issues at https://mypy.readthedocs.io/en/stable/runtime_troubles.html

[OddBloke]

Thanks for the conversation, @AlexWaygood, I appreciate it! Fully agreed on closing this out.

[Akuli]

all I'm doing is passing the return value of ArgumentParser.add_subparsers into a typed function. It's actually only typing that requires me to treat it as anything other than an undocumented, private class, if you know what I mean.

This sounds like from __future__ import annotations would help. It's available on 3.7+, and it prevents the interpreter from evaluating the annotations.

Edit: Nevermind. I just realized that you wanted a type alias, and this is also explained in the mypy docs that were already linked.

[OddBloke]

all I'm doing is passing the return value of ArgumentParser.add_subparsers into a typed function. It's actually only typing that requires me to treat it as anything other than an undocumented, private class, if you know what I mean.

This sounds like from __future__ import annotations would help. It's available on 3.7+, and it prevents the interpreter from evaluating the annotations.

Edit: Nevermind. I just realized that you wanted a type alias, and this is also explained in the mypy docs that were already linked.

I actually had the same thought! Yeah, it does improve matters slightly (because I wouldn't need to stringify each individual annotation), but it doesn't help with the reuse case, as you say.