Attempt to fix 'TestCheckResourceIntegration' flaky test #914
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
alenkacz
commented
Oct 9, 2019
| @@ -38,6 +40,8 @@ func TestMain(m *testing.M) { | |||
| } | |||
|
|
|||
| func TestCheckResourceIntegration(t *testing.T) { | |||
| rand.Seed(time.Now().UnixNano()) | |||
There was a problem hiding this comment.
I think this is actually not necessary but worth the try 😆
zen-dog
reviewed
Oct 9, 2019
| @@ -47,7 +51,7 @@ func TestCheckResourceIntegration(t *testing.T) { | |||
| { | |||
| testName: "match object by labels, first in list matches", | |||
| actual: []runtime.Object{ | |||
| testutils.WithSpec(testutils.WithLabels(testutils.NewPod("aa", ""), map[string]string{ | |||
| testutils.WithSpec(testutils.WithLabels(testutils.NewPod("labels-match-pod", ""), map[string]string{ | |||
There was a problem hiding this comment.
Lol, where have I seen these "speaking pod names"? 🤔
There was a problem hiding this comment.
:D 🤔 anyway the error is that pod 'aa' already exists :D 🤷♀
There was a problem hiding this comment.
We should make all pod names properly unique. Something along the lines of:
func PodName(suiteName, testName, podName string) {
reg, err := regexp.Compile("[^a-zA-Z0-9]+")
if err != nil {
log.Fatal(err)
}
return strings.ToLower(reg.ReplaceAllString(fmt.Sprintf("%s-%s-%s", suiteName, testName, podName), ""))
}and call it:
testutils.NewPod(PodName(t.Name, testName, "labels-match-pod"))There was a problem hiding this comment.
@zen-dog I kind of like them hand-picked based on the name of the test... it makes it easier for debugging as long as you're not generating 50 pods or so per test...
Meaning I prefer aa-my-test-with-labels and bb-my-test-with-labels where I can then look into code and see which one is which than fhafgag-my-test-with-labels and kafhgafg-my-test-with-labels not knowing which one is which :)
jbarrick-mesosphere
pushed a commit
that referenced
this pull request
Oct 17, 2019
* Introduce integration test for restart (#718) This introduces and integration test that: - creates instance - waits for deploy on that instance is finished - stops controller - updates instance CRD in the meantime - starts controller - verifies update plan was run This test was failing prior to the instance controller refactoring (the old controller logic was handling this wrong) * Fix namespace handling for plan command (#925) * Include CRDs in the -o yaml --dry-run (#923) * go mod changes makes repo dirty. this resolves that (#926) * Fix deep copy and don't modify original state (#919) * Fixes the release workflow for Docker images * Attempt to fix 'TestCheckResourceIntegration' flaky test (#914) * Do not use strategicmergepatch for CRDs (#936) * Plan status should show last executed plan, not just active plan (#931) * More consistent logging for install (#933) * fixing repo merge by name bug (#938) * Use built in isExists (#942) * Make it possible to opt out of junit (#950) * Maintainers is array, not object (#930) * Handle already exists in flaky test (#941) * Cleanup lastexecutedplan method to remove the flake (#949) * Fix docs for instance name in install (#943) * add namespace flag for kudo init (#903) * Fix 'run_tests' script * Use a shell script for E2E test commands The script is similar to the one used for integration tests. This also allows to not use JUnit output. * Copy E2E test script into test container * Fix permissions of E2E test script * Distinguish between local and operator E2E tests Tests in the 'test/e2e' folder are run as part of the E2E tests in addition to the operator tests.
jbarrick-mesosphere
added a commit
that referenced
this pull request
Oct 17, 2019
fix make docker privs add go-junit-report add docker CLI to test dockerfile fix docker cli install fix docker run command fix docker fix networking remove CMD from dockerfile E2E test additions (#958) * Introduce integration test for restart (#718) This introduces and integration test that: - creates instance - waits for deploy on that instance is finished - stops controller - updates instance CRD in the meantime - starts controller - verifies update plan was run This test was failing prior to the instance controller refactoring (the old controller logic was handling this wrong) * Fix namespace handling for plan command (#925) * Include CRDs in the -o yaml --dry-run (#923) * go mod changes makes repo dirty. this resolves that (#926) * Fix deep copy and don't modify original state (#919) * Fixes the release workflow for Docker images * Attempt to fix 'TestCheckResourceIntegration' flaky test (#914) * Do not use strategicmergepatch for CRDs (#936) * Plan status should show last executed plan, not just active plan (#931) * More consistent logging for install (#933) * fixing repo merge by name bug (#938) * Use built in isExists (#942) * Make it possible to opt out of junit (#950) * Maintainers is array, not object (#930) * Handle already exists in flaky test (#941) * Cleanup lastexecutedplan method to remove the flake (#949) * Fix docs for instance name in install (#943) * add namespace flag for kudo init (#903) * Fix 'run_tests' script * Use a shell script for E2E test commands The script is similar to the one used for integration tests. This also allows to not use JUnit output. * Copy E2E test script into test container * Fix permissions of E2E test script * Distinguish between local and operator E2E tests Tests in the 'test/e2e' folder are run as part of the E2E tests in addition to the operator tests.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
I spent quite some time investigating this flake and I must admit my defeat :) All my hunches proved not to be true. Anyway it looks like all the tests run correctly in their own namespace, I was not able to reproduce locally.
The only thing that looked suspicious was that petname generator library uses the same seed (but the default source should be safe for concurrent use so that looks right).
So this PR is just an attempt. It renames the other pod that was also named 'aa' in case those somehow end up running in the same namespace, even though I did not prove that to be true.
We'll see if this helped at all :)
Relates to #829