[codegen/python] - Implement dynamic config-getters

[komalali]

Description

This PR implements dynamic provider config getters for python. To avoid a breaking change, we define a class that extends the ModuleType type and implements property getters for each config key. We then overwrite the __class__ attribute of the current module as described in the proposal for PEP-549.

To continue to get autocomplete for the config keys, we now emit a __init__.pyi stub file in the config module which declares the variables and associated types.

I have added tests for the codegen side of things, but I'm struggling to figure out how to test the functionality in isolation. I used the following Automation API script to locally validate the result, but it requires a locally built pulumi_aws based on the codegen in this PR.

import pulumi
from pulumi import automation as auto
import pulumi_aws as aws

def pulumi_program():
    pulumi.export("region", aws.config.region)

project_name = "dynamic_config_test"
stack_name = "dev"
stack = auto.create_or_select_stack(stack_name=stack_name,
                                    project_name=project_name,
                                    program=pulumi_program)

stack.workspace.install_plugin("aws", "v4.11.0")

stack.set_config("aws:region", auto.ConfigValue(value="us-west-1"))
up_res = stack.up()
assert up_res.outputs["region"].value == "us-west-1"

stack.set_config("aws:region", auto.ConfigValue(value="us-east-1"))
up_res = stack.up()
assert up_res.outputs["region"].value == "us-east-1"

stack.destroy()

Python part of #5582

Notable Changes

• The actual codegen changes
• Generated vars.py
• Generated config/__init__.py
• Generated config/__init__.pyi

Checklist

• I have added tests that prove my fix is effective or that my feature works

• I have updated the CHANGELOG-PENDING file with my change

• Yes, there are changes in this PR that warrants bumping the Pulumi Service API version

[github-actions]

Diff for pulumi-random with merge commit 01d5fa1

[github-actions]

Diff for pulumi-azuread with merge commit 01d5fa1

[github-actions]

Diff for pulumi-random with merge commit 1c8a51e

[github-actions]

Diff for pulumi-azuread with merge commit 1c8a51e

[github-actions]

Diff for pulumi-gcp with merge commit 01d5fa1

[github-actions]

Diff for pulumi-gcp with merge commit 1c8a51e

[github-actions]

Diff for pulumi-azure with merge commit 1c8a51e

[github-actions]

Diff for pulumi-aws with merge commit 01d5fa1

[github-actions]

Diff for pulumi-aws with merge commit 1c8a51e

[github-actions]

Diff for pulumi-random with merge commit 2797193

[github-actions]

Diff for pulumi-random with merge commit b58eea8

[github-actions]

Diff for pulumi-random with merge commit a502b1e

[github-actions]

Diff for pulumi-random with merge commit 7b1a923

[github-actions]

Diff for pulumi-random with merge commit 63887d3

[justinvp]

Two initial comments after taking a quick look

[t0yv0]

🤯

[t0yv0]

I feel a little lost situating what this is for, do we have a ticket list? I'll try to read again next week with a fresh head. By the implementation of it, it sounds like schema can define config variables, types and defaults, and the PR makes these "type-safely" (I'd say statically) available to the autogenerated Python resource provider code. What's tripping me up is "dynamic", use of auto API, and what exactly are the bw-compat concerns.

[komalali]

@t0yv0 this is for the python part of #5582 - sorry, I had that in the PR description but I guess it got dropped at some point. The problem is defined pretty well in the issue, let me know if you need any more information.

The issue that @justinvp brought up about the types is somewhat orthogonal to the specific problem that this PR is solving but it came up because I tried to emit the output types and they are incorrect.

[github-actions]

Diff for pulumi-random with merge commit f107b64

[github-actions]

Diff for pulumi-random with merge commit 622c097

[justinvp]

LGTM otherwise.

It might be worth adding a comment that explains briefly why we're generating the _ExportableConfig and using sys.modules[__name__].__class__ = _ExportableConfig, and why we're generating the stub file.