fix: Generic definition

[mantasu]

Unless there is a specific reason to use _SpecialForm annotation, I'm proposing a slightly more refined way to define Generic:

• From Python 3.12, it is internal so we often fall back to the definition here - better to have something more accurate
• Types derived from _SpecialForm semantically prohibit subclassing whereas Generic[...] can be subclassed
• This benefits other types too (e.g., Protocol if in the future its definition is also updated from _SpecialForm)

Based on cpython spec, Generic has a custom __class_getitem__ that should be in the class template:

>>> set(dir(Generic)) - set(dir(object))
{'__module__', '__class_getitem__'}

[srittau]

I haven't looked at the changes in depth, but the fact that type checkers seem happy with it and the primer output indicates that this change has a positive impact look promising. But I'd like defer this decision to maintainers with more knowledge of typing internals. Maybe @JelleZijlstra or @AlexWaygood?

[AlexWaygood]

We'd have to make changes to ty if this change was made in typeshed; it causes quite a few of ty's tests to start failing if you just naively apply this patch to ty's vendored copy of typeshed:

failures:
    mdtest__annotations_self
    mdtest__annotations_stdlib_typing_aliases
    mdtest__annotations_unsupported_special_forms
    mdtest__async
    mdtest__attributes
    mdtest__call_function
    mdtest__call_methods
    mdtest__call_overloads
    mdtest__class_super
    mdtest__comprehensions_basic
    mdtest__dataclasses_dataclasses
    mdtest__decorators
    mdtest__directives_assert_never
    mdtest__directives_cast
    mdtest__exception_except_star
    mdtest__expression_lambda
    mdtest__expression_yield_and_yield_from
    mdtest__function_parameters
    mdtest__generics_legacy_classes
    mdtest__generics_legacy_functions
    mdtest__generics_legacy_variables
    mdtest__generics_legacy_variance
    mdtest__generics_pep695_classes
    mdtest__generics_scoping
    mdtest__ide_support_all_members
    mdtest__intersection_types
    mdtest__literal_collections_dictionary
    mdtest__loops_async_for
    mdtest__mdtest_config
    mdtest__mro
    mdtest__named_tuple
    mdtest__narrow_assignment
    mdtest__narrow_complex_target
    mdtest__narrow_type_guards
    mdtest__protocols
    mdtest__scopes_moduletype_attrs
    mdtest__subscript_bytes
    mdtest__subscript_lists
    mdtest__subscript_string
    mdtest__subscript_tuple
    mdtest__sys_version_info
    mdtest__type_of_basic
    mdtest__type_properties_is_assignable_to
    mdtest__type_properties_is_disjoint_from
    mdtest__type_properties_is_subtype_of
    mdtest__type_properties_materialization
    mdtest__typed_dict
    mdtest__with_async

The patch to fix ty after making this change isn't very complicated, though, so I wouldn't let that block that PR if it brings typeshed closer to the runtime semantics here:

diff --git a/crates/ty_python_semantic/src/types/infer.rs b/crates/ty_python_semantic/src/types/infer.rs
index a1a3cb2e7b..483d6493be 100644
--- a/crates/ty_python_semantic/src/types/infer.rs
+++ b/crates/ty_python_semantic/src/types/infer.rs
@@ -4667,20 +4667,11 @@ impl<'db, 'ast> TypeInferenceBuilder<'db, 'ast> {
         }
 
         // Handle various singletons.
-        if let Type::NominalInstance(instance) = declared.inner_type() {
-            if instance
-                .class(self.db())
-                .is_known(self.db(), KnownClass::SpecialForm)
+        if let Some(name_expr) = target.as_name_expr() {
+            if let Some(special_form) =
+                SpecialFormType::try_from_file_and_name(self.db(), self.file(), &name_expr.id)
             {
-                if let Some(name_expr) = target.as_name_expr() {
-                    if let Some(special_form) = SpecialFormType::try_from_file_and_name(
-                        self.db(),
-                        self.file(),
-                        &name_expr.id,
-                    ) {
-                        declared.inner = Type::SpecialForm(special_form);
-                    }
-                }
+                declared.inner = Type::SpecialForm(special_form);
             }
         }

Ideally I feel like we'd just say that Generic is a class, not a variable that could theoretically be any subclass of a private _Generic type. It's a shame that pyright doesn't seem to support that; this PR gets us closer to the runtime semantics but it would be nice to emulate them exactly.

Looking at the primer report:

• beartype (https://github.com/beartype/beartype/blob/0e31257efbb934c9a0160c54cd07497838ee4203/beartype/_util/hint/pep/proposal/pep484/pep484generic.py#L149 and https://github.com/beartype/beartype/blob/0e31257efbb934c9a0160c54cd07497838ee4203/beartype/_util/hint/pep/proposal/pep484/pep484generic.py#L177) -- these are false positives going away. Generic is a class but type checkers don't currently recognize it as being an instance of type because of our current annotations in typeshed. beartype is doing is_type_subclass(hint, Generic) where is_type_subclass expects either a type or a tuple of types as its second argument
• pydantic, strawberry, discord: these look like the same thing as beartype

It definitely seems like a win that this change means that type checkers understand that Generic is an instance of type out of the box -- that could be fixed with some special casing in ty/mypy, but it's nice to rely on typeshed where possible and implement the minimum required amount of special casing for this kind of thing in type checkers. So I'm +0.5 on this change.

The only last thing I'd like to check -- I'll kick off a draft PR on ty's CI to see if the ty diff I posted above has any performance implications for us.

[JelleZijlstra]

Yes, this seems surprisingly low-impact.

I don't see why there has to be a change between pre-3.12 and post-3.12 version. typing.Generic mostly works the same before and after 3.12, it's just implemented in C in 3.12.

[AlexWaygood]

I kicked off an experimental ty PR here to see if it has any impact on our mypy_primer report for ty users, or any performance impact for us: astral-sh/ruff#19969

[JelleZijlstra]

Why not just class Generic: without the type[] part? It's a class at runtime.

[mantasu]

Unfortunately, pyright was complaining a lot without this workaround 😕

[JelleZijlstra]

I don't see why there has to be a change between pre-3.12 and post-3.12 version. typing.Generic mostly works the same before and after 3.12, it's just implemented in C in 3.12.

I think I misremembered from a previous version of this PR, the current version looks good in that regard.

[AlexWaygood]

ty's CI does report a 5% perf regression as a result of the changes we'd have to make to accomodate this change, but only on one microbenchmark that's really there just to check that we don't have truly pathological performance on code that's very badly written.

The primer report on the typeshed PR indicates that this wouldn't have the same benefits right now to ty users that it has to mypy users. But I think that's just because ty doesn't really type-check calls to issubclass() at all right now (typeshed uses type aliases for the annotations in its issubclass() definition, and ty only supports very trivial type aliases right now).

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

beartype (https://github.com/beartype/beartype)
+ beartype/_util/hint/pep/proposal/pep484/pep484generic.py:149: error: Unused "type: ignore" comment  [unused-ignore]
+ beartype/_util/hint/pep/proposal/pep484/pep484generic.py:177: error: Unused "type: ignore" comment  [unused-ignore]

pydantic (https://github.com/pydantic/pydantic)
- pydantic/main.py:580: error: Argument 2 to "issubclass" has incompatible type "<typing special form>"; expected "_ClassInfo"  [arg-type]
- pydantic/dataclasses.py:203: error: Argument 2 to "issubclass" has incompatible type "<typing special form>"; expected "_ClassInfo"  [arg-type]
- pydantic/dataclasses.py:205: error: Incompatible types in assignment (expression has type "tuple[type[DataclassInstance], <typing special form>]", variable has type "tuple[type[DataclassInstance]]")  [assignment]
+ pydantic/dataclasses.py:205: error: Incompatible types in assignment (expression has type "tuple[type[DataclassInstance], Any]", variable has type "tuple[type[DataclassInstance]]")  [assignment]

strawberry (https://github.com/strawberry-graphql/strawberry)
+ strawberry/utils/typing.py:135: error: Unused "type: ignore" comment  [unused-ignore]
+ strawberry/utils/typing.py:185: error: Unused "type: ignore" comment  [unused-ignore]

discord.py (https://github.com/Rapptz/discord.py)
- ...typeshed_to_test/stdlib/typing.pyi:1016: note: "update" of "TypedDict" defined here
+ ...typeshed_to_test/stdlib/typing.pyi:1029: note: "update" of "TypedDict" defined here
- ...typeshed_to_test/stdlib/typing.pyi:1016: note: "update" of "TypedDict" defined here
+ ...typeshed_to_test/stdlib/typing.pyi:1029: note: "update" of "TypedDict" defined here
- discord/ext/commands/converter.py:1280: error: Argument 2 to "issubclass" has incompatible type "<typing special form>"; expected "_ClassInfo"  [arg-type]
- discord/ext/commands/hybrid.py:508: error: Overlap between argument names and ** TypedDict items: "name", "description"  [misc]
+ discord/ext/commands/hybrid.py:508: error: Overlap between argument names and ** TypedDict items: "description", "name"  [misc]
- discord/ext/commands/hybrid.py:834: error: Overlap between argument names and ** TypedDict items: "with_app_command", "name"  [misc]
+ discord/ext/commands/hybrid.py:834: error: Overlap between argument names and ** TypedDict items: "name", "with_app_command"  [misc]
- discord/ext/commands/hybrid.py:858: error: Overlap between argument names and ** TypedDict items: "with_app_command", "name"  [misc]
+ discord/ext/commands/hybrid.py:858: error: Overlap between argument names and ** TypedDict items: "name", "with_app_command"  [misc]
- discord/ext/commands/bot.py:290: error: Overlap between argument names and ** TypedDict items: "with_app_command", "name"  [misc]
+ discord/ext/commands/bot.py:290: error: Overlap between argument names and ** TypedDict items: "name", "with_app_command"  [misc]
- discord/ext/commands/bot.py:314: error: Overlap between argument names and ** TypedDict items: "with_app_command", "name"  [misc]
+ discord/ext/commands/bot.py:314: error: Overlap between argument names and ** TypedDict items: "name", "with_app_command"  [misc]