Test stack upgrades

[t0yv0]

Hello!

• Vote on this issue by adding a 👍 reaction
• If you want to implement this feature, comment to let us know (we'll work with you on design, scheduling, etc.)

Issue details

We should consider testing for no-changes provider upgrades.

There is a growing number of issues with provider upgrades generating unexpected changes in the users's stacks:

• 4.9.0 regenerates passwords pulumi-random#258 P1
• Updating cluster version from 1.21 -> 1.22 throws TypeError: Cannot read properties of null (reading 'data') on random previews. pulumi-eks#851
• Detecting a diff in GCP pubsub logging config pulumi-fastly#173

Some of these issues could have been detected by test automation checking specifically for the upgrade scenario. We currently do not know of an example of such automation.

• add pulumi-random test for stack upgrades (post mortem follow-up)
• discuss design and see anything can be upstreamed to ProgramTest
• figure out a plan for introducing this testing across other providers

Affected area/feature

Provider testing.

[t0yv0]

Implemented for Random provider: pulumi/pulumi-random#263

[t0yv0]

Given difficulties implementing this for random, this is now feature request against the ProgramTest framework.

pulumi/pulumi#12125

Once there's good options in the framework I recommend we circle back and ensure every one of our providers is covering the upgrade scenario for common resources with at least 1 test.

[t0yv0]

Plus one P1 on this, will be propagating the ad-hoc test to TLS.

[t0yv0]

Closing in favor of #325

[pulumi-bot]

Cannot close issue:

• does not have required labels: resolution/

Please fix these problems and try again.