Typing Feature Flags

[hmc-cs-mdrissi]

This idea is similar to stricter stubs idea, but a bit more general and focused more on new typing features. Typing features are produced at pretty quick pace. We're currently producing several typing peps each year. Paramspec, dataclasses, variadic, literal string, positional only, and so on. It looks like pace of a couple typing peps will continue for awhile.

We currently have multiple major type checkers. As new typing features are proposed, support for those features will appear in each type checker at different times. For typeshed and libraries that want to be typed and could gain type safety from using newer features, there's a problem of using too new feature may cause some user's experience to worsen if the type checker they use lacks support for that feature. So currently it looks like typeshed has a checklist of PEP accepted + support by each major type checker.

My proposal is intruding feature flags corresponding to major new typing features. We could have several bool constants like,

class TypingFeatures:
  PARAMSPEC_SUPPORTED = ...
  VARIADIC_SUPPORTED = ...
  LITERAL_STRING_SUPPORTED = ...

in typing.py/typing_extensions.py and then both stubs/libraries could use,

if TypingFeatures.PARAMSPEC_SUPPORTED:
  def contextmanager(func: Callable[_P, Iterator[_T]]) -> Callable[_P, ContextManager[_T]]: ...
else:
  def contextmanager(func: Callable[..., Iterator[_T]]) -> Callable[..., ContextManager[_T]]: ...

shortly after PEP/typing feature is accepted. The semantics of TypingFeatures.PARAMSPEC_SUPPORTED would be like TYPE_CHECKING except each type checker will define each feature as True/False.

I think this mainly applies to typing features that allow additional safety/make system more powerful. Features that only improve readability while important would not have value with a feature flag because while,

alias1 = int | str

is better then,

alias1 = Union[int, str]

but using a feature flag here makes it even less readable for no gain,

if TypingFeatures.UNION_OPERATOR:
  alias1 = int | str
else:
  alias1 = Union[int, str]

I'm mostly thinking of this for new/recent PEPs. The one other feature I'd be interested in a flag for is recursive types as it's very popular requested feature that different type checkers have different support for.

[JelleZijlstra]

I don't think this will work well in practice. "Support" is usually not a binary switch. For example, mypy added support for PEP 604 unions a long time ago, but it still has bugs in a few edge cases. Similarly, mypy added support for part of PEP 612 in one release, but hasn't released support for another part (Concatenate) yet. I expect PEP 646 support will also come gradually.

[hmc-cs-mdrissi]

Hmm, that sounds similar to how typeshed has to handle this. Using PEP 612 as example it is currently not marked for mypy because of Concatenate. Each of the issues for pep status has a checkmark list and only after feature is considered sufficiently implemented does it get checked. Although pytype's status of allowing it but not understanding it I'd probably not checkmark if we had ability to have stub for both cases. If a type checker has partial support for pep and support is not enough for common cases I would expect it to continue to be False. But I think each type checker is fully in control of when they consider there implementation complete enough. If an individual type checker wants to be more conservative about setting the flag to True they can.

Also main alternative for this is pretty much fork stubs and have different stubs per type checker. My team's codebase currently uses stubs that rely on some pyright specific features and if I upstream them I'll end up maintaining two variants. One variant that works across type checkers and second for pyright. This is doable, I just view it as a worse trade off then being able to mark individual aliases/functions with corresponding flag.

[davidfstr]

Also main alternative for this is pretty much fork stubs and have different stubs per type checker. My team's codebase currently uses stubs that rely on some pyright specific features and if I upstream them I'll end up maintaining two variants. One variant that works across type checkers and second for pyright. This is doable, I just view it as a worse trade off then being able to mark individual aliases/functions with corresponding flag.

A possible alternative to adding Feature Flags would be to add a way to detect the identity of the typechecker being used and use special stubs for typecheckers that haven't added support for a desired feature. For example:

if typing_extensions.TYPE_CHECKER in ['pytype']:  # does not support TypeGuard at all
    def isassignable(value: object, tp: Type[_T], *, eval: bool = True) -> bool:
        ...
else:
    def isassignable(value: object, tp: Type[_T], *, eval: bool = True) -> TypeGuard[_T]:
        ...

The above makes use of a proposed typing_extensions.TYPE_CHECKER constant that contains a string name of the type checker in use or '' (empty string) if no type checker is being used.

[TeamSpen210]

Feature flags are much better than identifying the type checker, we can learn from the web development community what a mess the USER_AGENT string became. In that hypothetical, once Pytype updated to support TypeGuard, you'd have to update the stub to follow suit (and therefore become incompatible with older versions). Instead with feature flags both the flag and support would occur simultaneously. The flags definitely need to be in their own namespace, so checkers can assume any unknown name is False and not an error.