Implement GetRequiredPlugins for Python

[justinvp]

This PR implements GetRequiredPlugins for Python, which is called by the engine at the beginning of pulumi preview|up to determine/install any missing plugins needed by the program.

This is implemented by running python -m pip list --format json to determine the installed Pulumi packages (packages with names that start with pulumi-). If the virtualenv runtime option is set in Pulumi.yaml, it runs the python inside the virtual environment. Otherwise, it calls python in the usual ways, which works with other tools that manage their own virtual environment (such as poetry run pulumi up).

The required plugins are determined based on the Pulumi packages. If a package contains an optional pulumiplugin.json file, the information in this file is used to determine the associated plugin: the name, version, and server. There's also a resource option that must be set to true to indicate this package has an associated plugin, or false to indicate the package does not have an associated plugin (e.g. I'd imagine we'd set this to false for pulumi-policy, in the meantime we hardcode this package as a known package that does not have a plugin). This file is similar to information included in package.json for Node.js and version.txt in .NET NuGet packages (though, version.txt does not currently support specifying a server). If the pulumiplugin.json file doesn't exist, we attempt to determine the plugin name and version based on the package name and version. This works well for all released packages we have today (which don't have a pulumiplugin.json yet), but does not work for alpha/beta/rc/dev builds of these packages because the package version does not include enough information to determine the plugin version for these cases. A commit is included with this PR, which adds support for generating the pulumiplugin.json file for packages (via a new schema option). I suspect we'll want to roll this out for all our packages, so GetRequiredPlugins will work for dev builds.

This PR also implements support for automatically creating the virtual environment and installing dependencies into it at the beginning of pulumi preview|up if the virtualenv option is set in Pulumi.yaml and the specified directory is missing or empty.

I'd love to add more automated testing around this, but may have to do that as a follow-up item (I don't believe we have much automated testing around similar functionality for the other languages).

Fixes #4848
Fixes #5164
Fixes #5593

[github-actions]

Diff for pulumi-random with merge commit 43610a1

[github-actions]

Diff for pulumi-azuread with merge commit 43610a1

[github-actions]

Diff for pulumi-kubernetes with merge commit 43610a1

[github-actions]

Diff for pulumi-gcp with merge commit 43610a1

[github-actions]

Diff for pulumi-azure with merge commit 43610a1

[github-actions]

Diff for pulumi-aws with merge commit 43610a1

[lukehoban]

This file is similar to information included in package.json for Node.js and version.txt in .NET NuGet packages (though, version.txt does not currently support specifying a server).

Should we be considering this as a standard format to use across these different language ecosystems?

[justinvp]

Should we be considering this as a standard format to use across these different language ecosystems?

We certainly could consider it.

• Node.js probably doesn't need it as there's already a pulumi section in package.json where this information lives.
• .NET has version.txt which indicates the NuGet package has a plugin. The file contains the plugin version (and also now the name of the plugin). We could consider further extending version.txt to include additional information like the plugin server, or consider moving .NET to pulumiplugin.json.
• Go doesn't currently have anything like this. It does a best effort of determining the plugin information based on the go modules. There isn't currently a way to specify a custom server for the plugin or explicit name or version. I suspect we could adopt pulumiplugin.json for Go.

[lukehoban]

Node.js probably doesn't need it as there's already a pulumi section in package.json where this information lives.

I guess I wonder if we should at least standardize on the JSON payload that we include inside the package.json, so that we have a consistent data model for this, even if it lives in different places for different package managers. Probably not a huge deal - but it feels like consistency here will help avoid capability drift between language ecosystems.

[justinvp]

standardize on the JSON payload that we include inside the package.json

I went ahead and did this. I defined a common PulumiPluginJSON struct that's used by both Python and now Node.js.

[github-actions]

Diff for pulumi-random with merge commit 552a8d9

[github-actions]

Diff for pulumi-azuread with merge commit 552a8d9

[github-actions]

Diff for pulumi-kubernetes with merge commit 552a8d9

[github-actions]

Diff for pulumi-gcp with merge commit 552a8d9

[github-actions]

Diff for pulumi-azure with merge commit 552a8d9

[github-actions]

Diff for pulumi-aws with merge commit 552a8d9

[github-actions]

Diff for pulumi-azuread with merge commit 160dd40

[github-actions]

Diff for pulumi-random with merge commit 160dd40

[github-actions]

Diff for pulumi-kubernetes with merge commit 160dd40

[github-actions]

Diff for pulumi-gcp with merge commit 160dd40

[github-actions]

Diff for pulumi-azure with merge commit 160dd40

[github-actions]

Diff for pulumi-aws with merge commit 160dd40