Implement support for closed TypedDicts (PEP 728)#21382
Open
alicederyn wants to merge 25 commits intopython:masterfrom
Open
Implement support for closed TypedDicts (PEP 728)#21382alicederyn wants to merge 25 commits intopython:masterfrom
alicederyn wants to merge 25 commits intopython:masterfrom
Conversation
PEP 728 allows multiple special forms to be applied to a key when a TypedDict is created using functional syntax, but the logic was only checking and stripping a single instance. This prevents overriding the overall total status on a ReadOnly field.
Fix a bug where a non-required key was considered consistent with a required key if the latter was marked as read-only.
Fix a bug where the meet of a mutable and readonly key was incorrectly set as readonly. This bug was caused by mistakenly unioning the readonly key sets, which happened to work for the tested case where readonly keys never appeared in the other type.
The TypedDict meet is returning Never whenever inputs have mismatched keys (value types or requiredness). However: * when a key is readonly, the meet value type can be a subtype of it * when a key is readonly and not required, the meet key can be required * when both keys are readonly and not required, the meet value type can be uninhabited (absent)
A typo was causing the readonly state of keys on the LHS of a TypedDict join to be ignored.
alicederyn
force-pushed
the
closed.keyword
branch
from
bd9fee9 to
34df265
Compare
This comment has been minimized.
This comment has been minimized.
Improve TypedDict joins where keys mismatch by returning a readonly key with the joined type instead of discarding it.
Prior to implementing support for subclass refinement, expand the test suite with all the cases that need to be tested.
Allow readonly keys to be refined in subclasses, in line with PEP 705. For cases where refinement is not permitted by spec, provide a more detailed error message. As placeholders may now affect validation without ending up in the final TypedDictType, an `analysis_incomplete` field has been added, triggering reanalysis in subsequent rounds. This closes python#7435
The fallback path in TypeAnalyser::visit_typeddict_type is copying the required keys from the original type, but not the readonly keys. This is resulting in incorrect subtyping analysis for type variables with TypedDict bounds with readonly keys.
`NotRequired[Never]` can be used to indicate that a single key in a TypedDict will not be present.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
Begin implementing PEP 728 support by adding an is_closed field to TypedDictType. This is filled out from the closed keyword, and displayed in reveal_type, but not otherwise supported yet.
Continue implementing PEP 728 support with updates to the subtyping logic. Drop the previous use of 'names_are_wider_than', which will be difficult to extend to cover `extra_items`, and instead use zipall to check for key addition/removal alongside the other key-based checks.
Propagate closed from subclasses. Verify that a subclass of a closed TypedDict does not add keys, nor override the closed status.
Close the join of two closed TypedDicts, and treat a missing key in a closed TypedDict as a `NotRequired[Never]` rather than the `ReadOnly[NotRequired[object]]` of an open TypedDict.
The meet of a closed TypedDict and another TypedDict must be closed. The implicit type of a missing key in a closed TypedDict is `ReadOnly[NotRequired[Never]]`.
Narrowing a union already treats final TypedDicts as closed; use the same logic if the closed keyword is used.
If a TypeVar with a TypedDict upper bound is encountered while narrowing, narrow based on the upper bound.
Propagate closed when analysing a TypedDictType with a fallback. This ensures subtype checks work correctly for a TypeVar with a closed TypedDict upper bound.
Unpacking a TypedDict with an undeclared key to a TypedDict with that key declared as not required is acceptable if the former is closed. Unpacking an open TypedDict into a closed TypedDict is never safe.
Close the kwargs TypedDict returned by ParamSpec where possible. Use an empty, closed TypedDict instead of a `dict[str, Never]` for functions that do not accept kwargs.
Contributor
|
Diff from mypy_primer, showing the effect of this PR on open source code: steam.py (https://github.com/Gobot1234/steam.py)
- steam/types/trade.py:69: error: Overwriting TypedDict field "instanceid" while merging [misc]
- steam/types/trade.py:69: error: Overwriting TypedDict field "classid" while merging [misc]
- steam/types/trade.py:112: error: Overwriting TypedDict field "assetid" while merging [misc]
- steam/types/trade.py:112: error: Overwriting TypedDict field "amount" while merging [misc]
- steam/types/trade.py:112: error: Overwriting TypedDict field "appid" while merging [misc]
- steam/types/trade.py:112: error: Overwriting TypedDict field "contextid" while merging [misc]
- steam/types/trade.py:112: error: Overwriting TypedDict field "instanceid" while merging [misc]
- steam/types/trade.py:112: error: Overwriting TypedDict field "classid" while merging [misc]
- steam/types/trade.py:112: error: Overwriting TypedDict field "missing" while merging [misc]
altair (https://github.com/vega/altair)
+ altair/vegalite/v6/schema/_config.py:6593: error: Unused "type: ignore" comment [unused-ignore]
+ altair/vegalite/v6/schema/_config.py:6618: error: Unused "type: ignore" comment [unused-ignore]
+ altair/vegalite/v6/api.py:699: error: Unused "type: ignore" comment [unused-ignore]
+ altair/vegalite/v6/api.py:725: error: Unused "type: ignore" comment [unused-ignore]
+ altair/vegalite/v6/api.py:774: error: Unused "type: ignore" comment [unused-ignore]
hydra-zen (https://github.com/mit-ll-responsible-ai/hydra-zen)
- src/hydra_zen/typing/_implementations.py:606: error: Overwriting TypedDict field "module" while extending [misc]
discord.py (https://github.com/Rapptz/discord.py)
- discord/types/emoji.py:42: error: Overwriting TypedDict field "animated" while extending [misc]
+ discord/types/scheduled_event.py:84: error: Field "user_count" is required in base class "_WithUserCount" but not in base class "StageInstanceScheduledEvent" [misc]
+ discord/types/scheduled_event.py:87: error: Field "user_count" is required in base class "_WithUserCount" but not in base class "VoiceScheduledEvent" [misc]
+ discord/types/scheduled_event.py:90: error: Field "user_count" is required in base class "_WithUserCount" but not in base class "ExternalScheduledEvent" [misc]
+ discord/types/interactions.py:214: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
+ discord/types/interactions.py:220: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
+ discord/types/interactions.py:226: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
+ discord/types/interactions.py:232: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
+ discord/types/interactions.py:234: error: Field "id" is not required and not readonly in base class "ComponentBase" [misc]
+ discord/types/interactions.py:239: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
+ discord/types/interactions.py:241: error: Field "id" is not required and not readonly in base class "ComponentBase" [misc]
+ discord/types/interactions.py:246: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
+ discord/types/interactions.py:248: error: Field "id" is not required and not readonly in base class "ComponentBase" [misc]
+ discord/types/interactions.py:268: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
+ discord/types/interactions.py:273: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
+ discord/types/components.py:54: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
+ discord/types/components.py:59: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
+ discord/types/components.py:87: error: Definition of field "type" incompatible with base class "SelectComponent" [misc]
+ discord/types/components.py:92: error: Definition of field "type" incompatible with base class "SelectComponent" [misc]
+ discord/types/components.py:97: error: Definition of field "type" incompatible with base class "SelectComponent" [misc]
+ discord/types/components.py:102: error: Definition of field "type" incompatible with base class "SelectComponent" [misc]
+ discord/types/components.py:107: error: Definition of field "type" incompatible with base class "SelectComponent" [misc]
+ discord/types/components.py:113: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
+ discord/types/components.py:125: error: Definition of field "type" incompatible with base class "SelectComponent" [misc]
+ discord/types/components.py:133: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
+ discord/types/components.py:139: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
+ discord/types/components.py:156: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
+ discord/types/components.py:169: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
+ discord/types/components.py:174: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
+ discord/types/components.py:182: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
+ discord/types/components.py:188: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
+ discord/types/components.py:195: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
+ discord/types/components.py:202: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
+ discord/types/components.py:210: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
+ discord/types/components.py:220: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
+ discord/types/components.py:232: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
- discord/types/scheduled_event.py:84: error: Overwriting TypedDict field "user_count" while merging [misc]
- discord/types/scheduled_event.py:87: error: Overwriting TypedDict field "user_count" while merging [misc]
- discord/types/scheduled_event.py:90: error: Overwriting TypedDict field "user_count" while merging [misc]
- discord/types/interactions.py:214: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/interactions.py:220: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/interactions.py:226: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/interactions.py:232: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/interactions.py:234: error: Overwriting TypedDict field "id" while extending [misc]
- discord/types/interactions.py:239: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/interactions.py:241: error: Overwriting TypedDict field "id" while extending [misc]
- discord/types/interactions.py:246: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/interactions.py:248: error: Overwriting TypedDict field "id" while extending [misc]
- discord/types/interactions.py:268: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/interactions.py:273: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/guild.py:142: error: Overwriting TypedDict field "stickers" while extending [misc]
- discord/types/components.py:54: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/components.py:59: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/components.py:87: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/components.py:92: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/components.py:97: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/components.py:102: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/components.py:107: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/components.py:113: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/components.py:125: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/components.py:133: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/components.py:139: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/components.py:156: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/components.py:169: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/components.py:174: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/components.py:182: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/components.py:188: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/components.py:195: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/components.py:202: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/components.py:210: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/components.py:220: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/components.py:232: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/gateway.py:180: error: Overwriting TypedDict field "newly_created" while extending [misc]
- discord/channel.py:144: error: Overwriting TypedDict field "bitrate" while extending [misc]
- discord/channel.py:145: error: Overwriting TypedDict field "user_limit" while extending [misc]
- discord/channel.py:146: error: Overwriting TypedDict field "rtc_region" while extending [misc]
- discord/channel.py:147: error: Overwriting TypedDict field "video_quality_mode" while extending [misc]
- discord/channel.py:148: error: Overwriting TypedDict field "overwrites" while extending [misc]
- discord/channel.py:151: error: Overwriting TypedDict field "topic" while extending [misc]
- discord/channel.py:152: error: Overwriting TypedDict field "slowmode_delay" while extending [misc]
- discord/channel.py:153: error: Overwriting TypedDict field "nsfw" while extending [misc]
- discord/channel.py:154: error: Overwriting TypedDict field "overwrites" while extending [misc]
- discord/channel.py:155: error: Overwriting TypedDict field "default_auto_archive_duration" while extending [misc]
- discord/channel.py:156: error: Overwriting TypedDict field "default_thread_slowmode_delay" while extending [misc]
+ discord/ext/commands/hybrid.py:61: error: Definition of field "description" incompatible with base class "_HybridCommandKwargs" [misc]
+ discord/ext/commands/hybrid.py:69: error: Definition of field "description" incompatible with base class "_HybridCommandDecoratorKwargs" [misc]
+ discord/ext/commands/hybrid.py:73: error: Definition of field "description" incompatible with base class "_HybridGroupKwargs" [misc]
- discord/ext/commands/hybrid.py:61: error: Overwriting TypedDict field "description" while extending [misc]
- discord/ext/commands/hybrid.py:64: error: Overwriting TypedDict field "with_app_command" while extending [misc]
- discord/ext/commands/hybrid.py:65: error: Overwriting TypedDict field "guild_ids" while extending [misc]
- discord/ext/commands/hybrid.py:66: error: Overwriting TypedDict field "guild_only" while extending [misc]
- discord/ext/commands/hybrid.py:67: error: Overwriting TypedDict field "default_permissions" while extending [misc]
- discord/ext/commands/hybrid.py:68: error: Overwriting TypedDict field "nsfw" while extending [misc]
- discord/ext/commands/hybrid.py:69: error: Overwriting TypedDict field "description" while extending [misc]
- discord/ext/commands/hybrid.py:73: error: Overwriting TypedDict field "description" while extending [misc]
- discord/ext/commands/bot.py:96: error: Overwriting TypedDict field "max_messages" while merging [misc]
- discord/ext/commands/bot.py:96: error: Overwriting TypedDict field "proxy" while merging [misc]
- discord/ext/commands/bot.py:96: error: Overwriting TypedDict field "proxy_auth" while merging [misc]
- discord/ext/commands/bot.py:96: error: Overwriting TypedDict field "shard_id" while merging [misc]
- discord/ext/commands/bot.py:96: error: Overwriting TypedDict field "shard_count" while merging [misc]
- discord/ext/commands/bot.py:96: error: Overwriting TypedDict field "application_id" while merging [misc]
- discord/ext/commands/bot.py:96: error: Overwriting TypedDict field "member_cache_flags" while merging [misc]
- discord/ext/commands/bot.py:96: error: Overwriting TypedDict field "chunk_guilds_at_startup" while merging [misc]
- discord/ext/commands/bot.py:96: error: Overwriting TypedDict field "status" while merging [misc]
- discord/ext/commands/bot.py:96: error: Overwriting TypedDict field "activity" while merging [misc]
- discord/ext/commands/bot.py:96: error: Overwriting TypedDict field "allowed_mentions" while merging [misc]
- discord/ext/commands/bot.py:96: error: Overwriting TypedDict field "heartbeat_timeout" while merging [misc]
- discord/ext/commands/bot.py:96: error: Overwriting TypedDict field "guild_ready_timeout" while merging [misc]
- discord/ext/commands/bot.py:96: error: Overwriting TypedDict field "assume_unsync_clock" while merging [misc]
- discord/ext/commands/bot.py:96: error: Overwriting TypedDict field "enable_debug_events" while merging [misc]
- discord/ext/commands/bot.py:96: error: Overwriting TypedDict field "enable_raw_presences" while merging [misc]
- discord/ext/commands/bot.py:96: error: Overwriting TypedDict field "http_trace" while merging [misc]
- discord/ext/commands/bot.py:96: error: Overwriting TypedDict field "max_ratelimit_timeout" while merging [misc]
- discord/ext/commands/bot.py:96: error: Overwriting TypedDict field "connector" while merging [misc]
pydantic (https://github.com/pydantic/pydantic)
- pydantic/fields.py:99: error: Unexpected keyword argument "closed" for "__init_subclass__" of "TypedDict" [call-arg]
|
Author
|
I'm not sure how exactly you'd prefer to have this change raised, so I've initially opened one big PR containing multiple small commits. Please LMK how you'd like it split up if that's the preference! |
Member
|
First of all thanks for working on this!
This may depend on who exactly will review this PR :-) Although this is a big PR, after quick look it seems quite non-controversial, so the review will be mostly double-checking all edge cases are handled correctly/consistently. So IMO this doesn't need splitting into parts. @JukkaL @hauntsaninja will any of you have time to look at this? (I am not sure if I will have time/energy to do a proper review, as I still have a backlog of NumPy-related stuff.) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implement support for the closed keyword on TypedDicts (part of PEP 728).
Additionally, fix some preexisting issues that I came across while updating the logic.
Overwriting TypedDict field "x" while merging#8714