[Epic] Python: Improved Typing

[justinvp]

For Python codegen we generate classes for inputs of object type to enable strong typing for these inputs. However, this leads to somewhat verbose syntax, requiring to instantiate the "args classes". See #11500 for a motivating example.

With PEP 589 – TypedDict and PEP 655 – Marking individual TypedDict items as required or potentially-missing there is now a more concise way to achieve the same result.

For each input object type we can generate a Args class as before, as well as a ArgsDict TypedDict based type. To support backwards compatibility, inputs of object type will accept the union of the two types.

class MyTypeArgsDict(TypedDict):
    my_prop: pulumi.Input[str]
    my_other_prop: NotRequired[pulumi.Input[float]]

Related issues: #11732

Example with Args

from pulumi_kubernetes.apps.v1 import Deployment
from pulumi_kubernetes.apps.v1 import Deployment, DeploymentSpecArgs
from pulumi_kubernetes.meta.v1 import LabelSelectorArgs, ObjectMetaArgs
from pulumi_kubernetes.core.v1 import ContainerArgs, PodSpecArgs, PodTemplateSpecArgs

app_labels = {"app": "nginx"}
deployment = Deployment(
    "nginx",
    spec=DeploymentSpecArgs(
        selector=LabelSelectorArgs(match_labels=app_labels),
        replicas=1,
        template=PodTemplateSpecArgs(
            metadata=ObjectMetaArgs(labels=app_labels),
            spec=PodSpecArgs(containers=[ContainerArgs(name="nginx", image="nginx")]),
        ),
    ),
)

Example with ArgsDict

from pulumi_kubernetes.apps.v1 import Deployment

app_labels = {"app": "nginx"}

deployment = Deployment(
    "nginx",
    spec={
        "selector": {"match_labels": app_labels},
        "replicas": 1,
        "template": {
            "metadata": {"labels": app_labels},
            "spec": {"containers": [{"name": "nginx", "image": "nginx"}]},
        },
    },
)

Design (M103)

• Design doc https://docs.google.com/document/d/1qNBzw-W1e80xhVpwQHNbwZ3hlfhMSBX-6TYF6UNcx0c/edit#heading=h.ojtxje7btj58
• Public copy https://docs.google.com/document/d/1fr1qAUE41mQq_YOmrP_aMncfFQXxU85jMpuOTeN9bYo/edit#heading=h.ojtxje7btj58

Implementation Plan

M104

• Get started on implementation
• Verify mypy/pyright/pycharm performance for TypedDict
• Review typing error messages for TypedDict

M106

• Generate TypedDicts for python inputs #15957
• Add new runtime options docs#12110
• Add inputTypes to schema language extensions for Python docs#12160

M107

• [sdk-gen/python] Generate non-plain object types being used as inputs for output-version invokes #14871
• Update registry docs - [discussion doc](Docs discussion https://docs.google.com/document/d/1BT4ILg-q-q-XgYRuDOBOYHdNIiICWcgpewMOhMdFcjE/edit#heading=h.ojtxje7btj58)
• Verify that generated docs use the new TypedDict types
• https://github.com/pulumi/pulumi.ai/issues/1011

Future Iteration

• Flip default to TypedDictSettingSideBySide #16375
• Add matrix tests for Python passing args and dicts #14088

Rollout & Announce

• Rollout to tier 1 providers
  • Add InputTypes flag to Python codegen information pulumi-terraform-bridge#2115
  • Enable TypedDict input types for the Python SDK pulumi-kubernetes#3070
  • Enable TypedDict input types for the Python SDK pulumi-aws#4108
  • Enable TypedDict input types for the Python SDK pulumi-gcp#2116
  • Enable TypedDict input types for the Python SDK pulumi-azure-native#3400
  • Enable TypedDict input types for the Python SDK pulumi-azure#2164
  • Enable TypedDict input types for the Python SDK pulumi-azuredevops#439
  • Enable TypedDict input types for the Python SDK pulumi-azuread#1180
• Update hand-written documentation examples to use the new TypedDict types
• Update templates to use new TypedDict types
• Blog post Python enhancements blog docs#12123
• 🍦

[jf]

hi Justin, for folks interested in the progress of this, where would we go to observe the progress and/or help with things?

[justinvp]

@jf, keeping an eye on this issue and #11732 is the best place to observe progress. When we have a fleshed out design proposal, we'll post it here.

[julienp]

We have a working implementation in #15957, however we ran into some performance issues when using MyPy ¹.

To work around this, we currently generate the TypedDict based args to behave differently depending on the used type checker. MyPy supports a special variable MYPY that is always considered true, and we can use this to hide the TypedDict types from it, and use Mapping[str, Any] instead. The PyDantic library currently uses a similar workaround, although for a different performance issue ².

Here's an example from the AWS provider:

if not MYPY:
    class InstanceCpuOptionsArgsDict(TypedDict):
        amd_sev_snp: NotRequired[pulumi.Input[str]]
        core_count: NotRequired[pulumi.Input[int]]
        threads_per_core: NotRequired[pulumi.Input[int]]
elif False:
    InstanceCpuOptionsArgsDict: TypeAlias = Mapping[str, Any]

When using Pyright, you get the full type checking experience within dictionary arguments, but not with MyPy.

The MyPy behaviour matches the previous implementation where these input types could either take an Args class or a Mapping[str, Any]. This means that we technically don't have a regression here, and when using Pyright you get a better experience.

However the question then becomes what style of arguments should we encourage? Currently we encourage the Args style classes over passing in dictionaries. It is the default for generated code and in documentation. This is not the most pythonic approach and fairly verbose ³, but does give strong typing and better IDE discoverability.

I'll also note that Pyright does not produce very readable error messages

Example

For the following code, Pyright correctly notes an error, but MyPy checking passes.

from pulumi_kubernetes.apps.v1 import Deployment

deployment = Deployment("nginx",
    spec={
        "selector":{
            "match_labels": { "app": "nginx" }
        },
        "replicas": "one", # This should be a Input[int]
        "template": { "spec": { "containers": [{ "name": "nginx", "image": "nginx" }] }
        }
    }
)

MyPy reports no issues

python3 -m mypy __main__.py
Success: no issues found in 1 source fil

Pyright reports an error, but the error message is not very friendly, and it is not obvious what the actual error is.

__main__.py:30:10 - error: Argument of type "dict[str, dict[str, dict[str, str]] | str | dict[str, dict[str, list[dict[str, str]]]]]" cannot be assigned to parameter "spec" of type "Input[DeploymentSpecArgs | DeploymentSpecArgsDict] | None" in function "__init__"
    Type "dict[str, dict[str, dict[str, str]] | str | dict[str, dict[str, list[dict[str, str]]]]]" is incompatible with type "Input[DeploymentSpecArgs | DeploymentSpecArgsDict] | None"
      "dict[str, dict[str, dict[str, str]] | str | dict[str, dict[str, list[dict[str, str]]]]]" is incompatible with "DeploymentSpecArgs"
      "dict[str, dict[str, dict[str, str]] | str | dict[str, dict[str, list[dict[str, str]]]]]" is incompatible with "DeploymentSpecArgsDict"
      "dict[str, dict[str, dict[str, str]] | str | dict[str, dict[str, list[dict[str, str]]]]]" is incompatible with protocol "Awaitable[DeploymentSpecArgs | DeploymentSpecArgsDict]"
        "__await__" is not present
      "dict[str, dict[str, dict[str, str]] | str | dict[str, dict[str, list[dict[str, str]]]]]" is incompatible with "Output[DeploymentSpecArgs | DeploymentSpecArgsDict]"
      "dict[str, dict[str, dict[str, str]] | str | dict[str, dict[str, list[dict[str, str]]]]]" is incompatible with "None" (reportArgumentType)
1 error, 0 warnings, 0 informations

Footnotes

1. https://github.com/python/mypy/issues/17231 ↩

2. https://github.com/pydantic/pydantic-core/pull/528 ↩

3. https://github.com/pulumi/pulumi/discussions/11500 ↩

[julienp]

It looks like PyCharm also picks Mapping[str, Any] when confronted with the if not MYPY/elif False workaround.

[justinvp]

Closing this out as fixed! 🎉

Blog announcement here: https://www.pulumi.com/blog/pulumi-loves-python/

There is some remaining work to rollout fully across the long tail of providers, docs, and examples, but we'll track the remaining work via the individual issues.