Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move upgrade tests to the new framework #3772

Merged
merged 4 commits into from Apr 3, 2024
Merged

Conversation

t0yv0
Copy link
Member

@t0yv0 t0yv0 commented Apr 2, 2024

This change makes tests consistent with the latest API exposed in github.com/pulumi/providertest/pulumitest and unblocks the upstream upgrade and release.

@t0yv0 t0yv0 mentioned this pull request Apr 2, 2024
Copy link

github-actions bot commented Apr 2, 2024

Does the PR have any schema changes?

Does the PR have any schema changes?

Looking good! No breaking changes found.
No new resources/functions.

Maintainer note: consult the runbook for dealing with any breaking changes.

Copy link

github-actions bot commented Apr 2, 2024

Is README.md missing any configuration options?

assumeRoleWithWebIdentity not found in Configuration section
customCaBundle not found in Configuration section
defaultTags not found in Configuration section
ec2MetadataServiceEndpoint not found in Configuration section
ec2MetadataServiceEndpointMode not found in Configuration section
httpProxy not found in Configuration section
httpsProxy not found in Configuration section
ignoreTags not found in Configuration section
noProxy not found in Configuration section
retryMode not found in Configuration section
s3UsEast1RegionalEndpoint not found in Configuration section
sharedConfigFiles not found in Configuration section
skipRequestingAccountId not found in Configuration section
stsRegion not found in Configuration section
tokenBucketRateLimiterCapacity not found in Configuration section
useDualstackEndpoint not found in Configuration section

Please add a description for each of these options to README.md. Details about them can be found in either the upstream docs or schema.json.

@t0yv0 t0yv0 marked this pull request as ready for review April 2, 2024 23:42
@t0yv0 t0yv0 requested a review from a team April 3, 2024 00:05
func TestQueue(t *testing.T) {
nodeTest(t, filepath.Join("..", "examples", "queue"),
providertest.WithSkippedUpgradeTestMode(providertest.UpgradeTestMode_Quick, "Prefer PreviewOnly"),
providertest.WithDiffValidation(providertest.NoReplacements()))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the default assert for all tests now "no replacements"? Non-blocking but it was previously "no changes" so it'd be nice to bring it back to that, otherwise we'd be losing coverage.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. That's intentional. Testing for absence of updates is not a great default for us since it's totally fine with the users in most of the cases but creates friction that makes it difficult to iterate in this repository. I think we need to be looking at stricter checks when it is meaningful to some end user-scenario not here. They would be interesting to have in some kind of statistical/advisory role but the test suite right now is basically pass or fail there is no easy way to track e.g. warnings.

@t0yv0 t0yv0 merged commit 2d5ad20 into master Apr 3, 2024
23 checks passed
@t0yv0 t0yv0 deleted the t0yv0/refactored-upgrade-tests branch April 3, 2024 12:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants