Python azure-native program fails when built with the latest codegen

[mikhailshilkov]

I'm updating azure-native's codegen from 3.19.0 to what's in master (commit 278055c) in this PR. After the update, the Python program stops working with this panic:

failed to discover plugin requirements: json: Unmarshal(nil *plugin.PulumiPluginJSON)

see https://github.com/pulumi/pulumi-azure-native/runs/4532362541?check_suite_focus=true#step:21:125

Do I need to change anything in the provider code?

[justinvp]

Well, shoot...

A while ago, I added support in the Python language host to read an optional pulumiplugin.json file which could optionally be included in a Python package to provide additional metadata about the associated Pulumi plugin. The Python codegen did not generate this file by default (you'd have to set a flag in the schema to opt-in) and none of our providers were using it. Now, we're moving ahead with using this file more broadly in Go and .NET as well (to better support custom server URLs) and will be generating it by default.

Unfortunately, there is a bug in how older CLIs unmarshal the json:

pulumi/sdk/go/common/resource/plugin/plugin.go

Lines 77 to 78 in 8ea38f6

	var plugin *PulumiPluginJSON
	if err := json.Unmarshal(b, plugin); err != nil {

This returns an error due to a nil value passed to json.Unmarshal.

@iwahbe fixed this to work correctly recently in #8515:

pulumi/sdk/go/common/resource/plugin/plugin.go

Lines 77 to 78 in ee81da2

	plugin := &PulumiPluginJSON{}
	if err := json.Unmarshal(b, plugin); err != nil {

Once we start including this file in all our Python packages by default, any existing/older CLI without the fix from #8515 is going to run into this error, which is not good. We're going to need to come up with a solution because we want older CLIs to continue to work.

I think the best way to avoid this problem is to use a different filename than pulumiplugin.json. That way, any older CLIs won't hit this path because the file won't exist. We could use a different filename for just Python, but since we haven't shipped this yet for the other languages, we could consider a new filename across all languages, which will be nicer for consistency. Maybe pulumi-plugin.json? Definitely open to better names or alternative ideas.

---

@mikhailshilkov, as an aside, the GHA workflow for azure-native is installing v3.0.0-rc.1 of the CLI. Any reason not to use a more recent version of the CLI? https://github.com/pulumi/pulumi-azure-native/blob/2f96c035be2064506b628a62a7f5d539d1f4fc81/.github/workflows/pull_request.yml#L59

[mikhailshilkov]

the GHA workflow for azure-native is installing v3.0.0-rc.1 of the CLI. Any reason not to use a more recent version of the CLI? https://github.com/pulumi/pulumi-azure-native/blob/2f96c035be2064506b628a62a7f5d539d1f4fc81/.github/workflows/pull_request.yml#L59

That's definitely an oversight, we'll fix it.

Is there an option to switch off this file generation for now?

[mikhailshilkov]

Or does the 3.19 CLI contain a fix for this already?

[justinvp]

Is there an option to switch off this file generation for now?

Unfortunately, no.

Or does the 3.19 CLI contain a fix for this already?

No, the fix from #8515 has not been released yet.

I know this isn't great, but you could temporarily workaround it by adding a post-codgen step in the Makefile for Python to delete the pulumiplugin.json and delete the 'pulumiplugin.json', line in setup.py.