-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix a flaky test #3335
Fix a flaky test #3335
Conversation
|
||
repeat(iterations) { | ||
while (eCnt == 0 || e1Cnt == 0 || e2Cnt == 0 || loopsToMake-- > 0) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The rationale behind this test was the following: given enough parallelism (3 threads) and iterations, we expect to observe every exception type at least once.
Taking into account recent changes in our CI and the nature of this test, I would suggest changing it to be more robust, but keep the original intent: iterate until all exception types are observed OR really many iterations passed (to address flakyness), but add a large enough timeout so CI won't hang for four hours
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Given the rationale, I don't understand the need to count iterations, as opposed to just doing this, with something like withTimeout
wrapping everything:
while (eCnt == 0 || e1Cnt == 0 || e2Cnt == 0 || loopsToMake-- > 0) { | |
while (eCnt == 0 || e1Cnt == 0 || e2Cnt == 0) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was thinking about iterations for fast debugging locally and a large enough timeout for slow CI, but this should work as well, up to you
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think not having iterations, in this case, is more semantically clear. Most other stress tests perform many iterations so that a problematic condition has more chances to arise. Here, it's not the case: we want to wait for a success condition. Waiting for a given number of iterations obscures this, whereas the concept of a timeout is clear and self-evident.
Not sure what the test checks for. Does it really need to make all these iterations, or just waiting for every type of exception to fire is enough?