Improve Python tooling experience for complex input/output shapes

[lukehoban]

We should strongly consider emitting TypedDicts for all dictionaries we use for inputs and outputs: https://www.python.org/dev/peps/pep-0589/.

This would provide some chance of decent tooling experiences for complex input/output shapes which are prevalent in Kubernetes.

We should do the same in tfgen as well, though the issue is incrementally less critical there.

We decided to generate classes rather than use TypedDicts.

Remaining tasks

• #4789: Support serializing/deserializing Python classes
• #5140: Follow-up: Instantiate output types for calls to invoke
• #5141: Follow-up: Automatically generate __init__ for input classes
• #5142: Follow-up: Support docstrings in IDE hover tooltips over output properties for resources and function results
• Pulumi SDK release that contains #4789 (Pulumi 2.9.0)
• #4790: Python codegen: Emit classes for nested inputs/outputs
• [In-Progress] Provider releases
• #4791: Python docgen: Refer to classes for nested inputs/outputs
• #4792: Python program codegen: Use classes when generating Python programs
• [In-Progress] docs/#3534: Update Python docs to use classes for nested inputs/outputs
• [In-Progress] examples/#700: Update Python examples to use classes
• [Done; Blocked on K8s provider release] templates/#120: Update Python templates to use classes
• [Available] Demos
• Blog post with screenshots/gifs and/or video

[hausdorff]

@lukehoban This is only supported starting in 3.8, a forthcoming release. Also vscode doesn't support it yet, and it's not clear when it will. We could maybe add support ourselves.

[lukehoban]

Python 3.8 is now available with this support.

We've also added mypy checking into the core and are starting to publish type information, so this will light up error checking.

I'll move this to pulumi/pulumi as it is something we'll want to do broadly across all SDKs.

[justinvp]

If we wanted to avoid bumping the min required version of Python to 3.8, we could consider using dataclasses (requires 3.7): https://docs.python.org/3/library/dataclasses.html. Or just generate the equivalent code. As part of this, we should loosen up any code specifically looking for dict to look for container-like types instead and have the dataclasses emulate dictionaries (https://docs.python.org/3/reference/datamodel.html#emulating-container-types).

[praneetloke]

This is also relevant for resource docs generator. See this comment as an example.

[leezen]

I've renamed the issue to reflect what we're really trying to do. The discussion from #4510 resulted in going forward with using dataclasses.

[justinvp]

Providers have been rolled out with these changes. There is some trailing docs and examples updates that are in-progress, tracked at pulumi/docs#3534 and pulumi/examples#700.