Generate TypedDicts for python inputs #15957

julienp · 2024-04-16T15:40:39Z

Epic: Improved Typing #12689

This PR adds the flag Languages.Python.InputTypes to the schema, which can take the values classes or classes-and-dicts. In the first iteration of the feature, this defaults to classes, which leaves code generation as is and does not change input types.. If the flag is set to classes-and-dicts, TypedDict based types are generated next to the current <Type>Args classes. In the future this could be extended to support dicts to generate only TypedDict types.

The generated types are conditional on the used type checker to work around perf issues in MyPy and PyCharm, see #12689 (comment)

if not MYPY:
    class DeploymentArgsDict(TypedDict):
        api_version: NotRequired[Input[str]]
        kind: NotRequired[Input[str]]
        metadata: NotRequired[Input['ObjectMetaArgsDict']]
        ...
elif False:
    DeploymentArgsDict: TypeAlias = Mapping[str, Any]

Removing the workaround is tracked in #16408

pulumi-bot · 2024-04-16T15:41:23Z

Changelog

[uncommitted] (2024-06-17)

Features

[sdkgen/python] Generate TypedDict types for inputs
#15957

tgummerer

Couple of minor suggestions inline, but overall this looks good to me!

tgummerer · 2024-06-17T09:50:47Z

pkg/codegen/python/gen.go

+		for key, value := range requires {
+			updatedRequires[key] = value
+		}
+		updatedRequires["typing-extensions"] = ">=4.11; python_version < \"3.11\""


Suggested change

updatedRequires["typing-extensions"] = ">=4.11; python_version < \"3.11\""

updatedRequires["typing-extensions"] = ">=4.11,<5; python_version < \"3.11\""

I think that would be safer in case there's a new major version release with breaking changes.

tgummerer · 2024-06-17T09:55:24Z

pkg/codegen/python/gen.go

+	indent := "    "
+	name = pythonCase(name)
+
+	// Running mypy gets very slow when there are a lot of TypedDicts.


Should this be a TODO with an open issue on our side linking to the upstream issue? Assuming it would be better to at some point also run this for mypy if the upstream issue is fixed?

Good idea! I created #16408 and linked via TODO

Tentative changelog: ### Features - [sdk/go] Suggest valid attributes with similar names to unrecognised ones when validating project definitions [#16097](#16097) - [cli/new] Allow passing runtime options as args in pulumi new [#16346](#16346) - [cli/new] Query language runtime for options during “pulumi new” [#16346](#16346) - [sdk/nodejs] Detect pnpm workspaces when running pulumi install [#15525](#15525) - [sdk/nodejs] Add options to Workspace::removeStack() [#16333](#16333) - [sdk/python] Automatically convert requirements.txt to pyproject.toml when using Poetry [#16346](#16346) - [sdkgen/python] Generate TypedDict types for inputs [#15957](#15957) ### Bug Fixes - [engine] Fix a panic when ignoring wildcard values with arrays of different length [#16406](#16406) - [cli/engine] Fix --continue-on-error running indefinitely when a resource fails to be created or updated [#16371](#16371) - [cli/plugin] Fix plugin install command when plugin type is tool [#16407](#16407) - [sdk/python] Don't incorrectly emit deprecation warnings for non-deprecated properties [#16400](#16400) - [sdk/python] Handle extra CLI arguments passed policy packs plugins [#16402](#16402) - [sdk/python] Add VIRTUAL_ENV environment variable when running inside a virtual environment ### Miscellaneous - [cli/new] Instruct the user to use 'pulumi install' when using --generate-only

Waiting on #16441, but there are also a few other fixes in the queue that I'll opportunistically wait on as well. Tentative changelog: ### Features - [engine] Resolve provider in the engine before passing it to transforms [#16409](#16409) - [sdk/go] Suggest valid attributes with similar names to unrecognised ones when validating project definitions [#16097](#16097) - [cli/new] Allow passing runtime options as args in pulumi new [#16346](#16346) - [cli/new] Query language runtime for options during “pulumi new” [#16346](#16346) - [cli/new] Add packagemanager prompt to pulumi new for nodejs [#16417](#16417) - [sdk/nodejs] Detect pnpm workspaces when running pulumi install [#15525](#15525) - [sdk/nodejs] Add options to Workspace::removeStack() [#16333](#16333) - [sdk/python] Automatically convert requirements.txt to pyproject.toml when using Poetry [#16346](#16346) - [sdkgen/python] Generate TypedDict types for inputs [#15957](#15957) ### Bug Fixes - [engine] Fix a panic when ignoring wildcard values with arrays of different length [#16406](#16406) - [cli/engine] Fix --continue-on-error running indefinitely when a resource fails to be created or updated [#16371](#16371) - [sdk/nodejs] Avoid an unhandled error when `dependencies` is missing from `package.json` during closure serialization [#16433](#16433) - [cli/plugin] Fix plugin install command when plugin type is tool [#16407](#16407) - [sdk/python] Fix Python SDK docs by escaping the trailing underscore in a docstring [#14866](#14866) - [sdk/python] Don't incorrectly emit deprecation warnings for non-deprecated properties [#16400](#16400) - [sdk/python] Handle extra CLI arguments passed policy packs plugins [#16402](#16402) - [sdk/python] Add VIRTUAL_ENV environment variable when running inside a virtual environment [#16425](#16425) - [sdk/python] Don't lift dunder attributes on `Output`s ### Miscellaneous - [sdk/{go,nodejs,python}] Add register resource transform alias for register stack transform - [cli/new] Instruct the user to use 'pulumi install' when using --generate-only [#16411](#16411)

This flag was added to pulumi Python codegen in pulumi/pulumi#15957 and enables the generation of TypedDict based input types. This is currently opt in and defaults to `classes`. A future version of Python codegen will flip this to `classes-and-dicts` to generate TypedDict types side by side with Args classes.

This flag was added to Pulumi Python codegen in pulumi/pulumi#15957 and enables the generation of TypedDict based input types. This is currently opt in and defaults to `classes`, leaving codegen unchanged. A future version of Python codegen will flip this to `classes-and-dicts` to generate TypedDict types side by side with Args classes.

pulumi-bot · 2024-06-24T17:40:17Z

This PR has been shipped in release v3.121.0.

julienp force-pushed the julienp/typeddict branch 2 times, most recently from 5520b90 to b95ed23 Compare April 16, 2024 15:50

julienp force-pushed the julienp/typeddict branch from b95ed23 to 47ebee5 Compare April 17, 2024 08:34

julienp added 11 commits June 13, 2024 14:05

Import sys so we can check the version

46b69e8

Fix typo in comment

a26a7f3

Generate TypedDict input types when the TypedDictArgs flag is set

b3497ef

Add test case for TypedDictArgs

4f1f1cd

Update tests

5745681

Use typeddict for arguments in construction expressions

3b6be8d

Add function with TypedDict types

930912a

Disable TypedDict for mypy

93ec3da

Handle empty dict definitions

74b39cc

Make TypedDictArgs a stringy enum

30f38e3

Only include typing-extensions when generating TypedDIcts

a638aaf

julienp force-pushed the julienp/typeddict branch from bd53138 to 62598d2 Compare June 13, 2024 12:05

Rename flag to InputTypes

843b95b

julienp force-pushed the julienp/typeddict branch from 62598d2 to 843b95b Compare June 13, 2024 13:22

tgummerer approved these changes Jun 17, 2024

View reviewed changes

julienp mentioned this pull request Jun 17, 2024

Remove MyPy workaround for TypedDicts #16408

Open

julienp added 2 commits June 17, 2024 14:22

tighten typing-extensions version range

06b35c7

add todo comment

7ddef9a

julienp added this pull request to the merge queue Jun 18, 2024

Merged via the queue into master with commit 05574aa Jun 18, 2024
55 of 61 checks passed

julienp deleted the julienp/typeddict branch June 18, 2024 16:12

justinvp mentioned this pull request Jun 20, 2024

Prepare for v3.121.0 release #16430

Merged

justinvp mentioned this pull request Jun 21, 2024

Freeze v3.121.0 #16447

Merged

julienp mentioned this pull request Jun 22, 2024

Add InputTypes flag to Python codegen information pulumi/pulumi-terraform-bridge#2115

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generate TypedDicts for python inputs #15957

Generate TypedDicts for python inputs #15957

julienp commented Apr 16, 2024 •

edited

Loading

pulumi-bot commented Apr 16, 2024 •

edited

Loading

tgummerer left a comment

tgummerer Jun 17, 2024

tgummerer Jun 17, 2024

julienp Jun 17, 2024

pulumi-bot commented Jun 24, 2024

	updatedRequires["typing-extensions"] = ">=4.11; python_version < \"3.11\""
	updatedRequires["typing-extensions"] = ">=4.11,<5; python_version < \"3.11\""

Generate TypedDicts for python inputs #15957

Generate TypedDicts for python inputs #15957

Conversation

julienp commented Apr 16, 2024 • edited Loading

pulumi-bot commented Apr 16, 2024 • edited Loading

Changelog

[uncommitted] (2024-06-17)

Features

tgummerer left a comment

Choose a reason for hiding this comment

tgummerer Jun 17, 2024

Choose a reason for hiding this comment

tgummerer Jun 17, 2024

Choose a reason for hiding this comment

julienp Jun 17, 2024

Choose a reason for hiding this comment

pulumi-bot commented Jun 24, 2024

julienp commented Apr 16, 2024 •

edited

Loading

pulumi-bot commented Apr 16, 2024 •

edited

Loading