Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

resource: Fix flaky test due to missing Done call #25646

Merged

Conversation

joamaki
Copy link
Contributor

@joamaki joamaki commented May 24, 2023

The workaround in 5fa2ac4 was faulty as it did not call ev.Done on the received event leading to further events not being received and thus eventually causing the context to time out. The problem was reproduced reliably locally by adding short time.Sleep calls to the pushUpdate and the initial listing to trigger the race causing the double updates.

To remedy this, add the missing ev.Done call, handle timeout gracefully and make assertions FailNow immediately to not mask problems due to e.g. nil deref.

FIxes: #24696
Fixes: 5fa2ac4 ("resource: Work around a rare race in initial sync")

@maintainer-s-little-helper maintainer-s-little-helper bot added the dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. label May 24, 2023
@joamaki joamaki force-pushed the pr/joamaki/fix-resource-test-flakyness branch from 51ad940 to 4cd6d56 Compare May 24, 2023 14:06
@@ -40,7 +41,7 @@ func TestMain(m *testing.M) {
goleak.VerifyTestMain(m, goleak.Cleanup(cleanup))
}

func testStore(t *testing.T, node *corev1.Node, store resource.Store[*corev1.Node]) {
func testStore(t testing.TB, node *corev1.Node, store resource.Store[*corev1.Node]) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: the change is probably not required anymore? Conventionally this should be tb now which would lead to a lot of churn.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reverted this hunk.

The workaround in 5fa2ac4 was faulty as it did not call ev.Done
on the received event leading to further events not being received
and thus eventually causing the context to time out. The problem was
reproduced reliably locally by adding short time.Sleep calls to the
pushUpdate and the initial listing to trigger the race causing the
double updates.

To remedy this, add the missing ev.Done call, handle timeout
gracefully and make assertions FailNow immediately to not mask
problems due to e.g. nil deref.

FIxes: cilium#24696
Fixes: 5fa2ac4 ("resource: Work around a rare race in initial sync")

Signed-off-by: Jussi Maki <jussi@isovalent.com>
@joamaki joamaki force-pushed the pr/joamaki/fix-resource-test-flakyness branch from 4cd6d56 to c8e75ad Compare May 24, 2023 14:10
@joamaki joamaki marked this pull request as ready for review May 29, 2023 13:31
@joamaki joamaki requested a review from a team as a code owner May 29, 2023 13:31
@joamaki joamaki requested a review from youngnick May 29, 2023 13:31
@joamaki joamaki added the release-note/misc This PR makes changes that have no direct user impact. label May 29, 2023
@maintainer-s-little-helper maintainer-s-little-helper bot removed the dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. label May 29, 2023
@joamaki
Copy link
Contributor Author

joamaki commented May 29, 2023

Relevant test is the pkg/k8s/resource unit test (run as part of Travis CI) and since there's no changes outside tests we don't need the full CI run.

@joamaki joamaki added the ready-to-merge This PR has passed all tests and received consensus from code owners to merge. label Jun 1, 2023
@julianwiedmann julianwiedmann merged commit e76d40f into cilium:main Jun 2, 2023
45 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ready-to-merge This PR has passed all tests and received consensus from code owners to merge. release-note/misc This PR makes changes that have no direct user impact.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

CI: TestResource_WithFakeClient panic
4 participants