ProgramTest should support testing provider upgrades

[t0yv0]

Hello!

• Vote on this issue by adding a 👍 reaction
• If you want to implement this feature, comment to let us know (we'll work with you on design, scheduling, etc.)

Issue details

When working with providers, it is important to test that a new version of the provider (combined with language SDK) will roll out cleanly and generate no-op previews and updates for existing stacks built using a provider version. I would be ideal if ProgramTest supported this.

As an example, pulumi/pulumi-random#258 was a P1 issue in pulumi-random that could have been trivially caught by a test of this kind. There is now a test included, but it requires a lot of woarkarounds over limitations of ProgramTest: pulumi/pulumi-random#263

See especially the comments in examples/internal/testutil/testutil.go

As part of working this out, it seems that RunUpdateTest option is not working also:

	RunUpdateTest bool

There's two problems with it:

• it is difficult to use with a local provider binary in PATH, it only tests the node SDK update

• if one works around that, it actually ignores meaningful replacements - the test passes when the provider erroneously generates the replace plans

Affected area/feature

[Frassle]

it is difficult to use with a local provider binary in PATH, it only tests the node SDK update

This is what LocalProviders option is for, and/or plugin section in the Pulumi.yaml file.

[t0yv0]

I'd appreciate some more information.

• LocalProviders is not documented; I've poked around at some integration tests that seem to point to folders with PulumiPlugin.yaml which appear undocumented also

• Plugin section in the Pulumi.yaml file - where is that documented, https://www.pulumi.com/docs/reference/yaml/ has no reference

• YAML is nice but legacy use cases are often TypeScript

• Even with these options, how can one shoe-horn ProgramTest to perform A/B testing on different versions of providers (A=prod, B=locally-built)

If you attempt to work out a full exercise it will highlight the pain points.

[Frassle]

LocalProviders is not documented;

I'll fix that, it's pretty much the same as the plugin section.

Plugin section in the Pulumi.yaml file

Wrong YAML page, you want https://www.pulumi.com/docs/reference/pulumi-yaml/#plugins-options. It's a general project option, not a yaml runtime option.

YAML is nice but legacy use cases are often TypeScript

As above the plugins option works for programs in any language not just yaml.

Even with these options, how can one shoe-horn ProgramTest to perform A/B testing on different versions of providers

That's a good question! I've been pondering that with @danielrbradley recently as well. I think this ties into https://github.com/pulumi/platform-providers-team/issues/89 where rather than building to a global go/bin if the provider just built to a local bin folder then all tests could just point there.
That could then match up in CI and locally, you just do the same thing maybe with CI pre-building and restoring the artifact in parallel test jobs.

@danielrbradley do you want to pair up quickly on a provider to see if we can do this? If we hit any issues I can fix them up in ProgramTest and then you can roll out to the other providers?

[t0yv0]

Ah thanks for the refs I'll read up on what's available.

[t0yv0]

https://github.com/pulumi/pulumi-random/blob/master/examples/examples_nodejs_test.go#L25 and especially https://github.com/pulumi/pulumi-random/blob/master/examples/internal/testutil/testutil.go#L36 for a reference of what we're doing now. Random actually does built into $PWD/bin. If you run this upgrade test on old=v4.8.2 new=v4.9.0 it should fail because 4.9.0 had a bug with incorrect replace plans.

[t0yv0]

Friel thought we can perhaps utilize "EditDirs" but without edit dirs. It seems that feature is more properly named "Steps". Something to consider - open to anything that fits the use case.

[t0yv0]

Possibly made easier by #13442 but does not strictly need that.

[t0yv0]

Related: pulumi/pulumi-aws#2682