New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Move upgrade tests to the new framework #3772
Conversation
Does the PR have any schema changes?Does the PR have any schema changes?Looking good! No breaking changes found. Maintainer note: consult the runbook for dealing with any breaking changes. |
Is README.md missing any configuration options?
Please add a description for each of these options to |
func TestQueue(t *testing.T) { | ||
nodeTest(t, filepath.Join("..", "examples", "queue"), | ||
providertest.WithSkippedUpgradeTestMode(providertest.UpgradeTestMode_Quick, "Prefer PreviewOnly"), | ||
providertest.WithDiffValidation(providertest.NoReplacements())) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is the default assert for all tests now "no replacements"? Non-blocking but it was previously "no changes" so it'd be nice to bring it back to that, otherwise we'd be losing coverage.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes. That's intentional. Testing for absence of updates is not a great default for us since it's totally fine with the users in most of the cases but creates friction that makes it difficult to iterate in this repository. I think we need to be looking at stricter checks when it is meaningful to some end user-scenario not here. They would be interesting to have in some kind of statistical/advisory role but the test suite right now is basically pass or fail there is no easy way to track e.g. warnings.
This change makes tests consistent with the latest API exposed in github.com/pulumi/providertest/pulumitest and unblocks the upstream upgrade and release.