Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix a flaky test #3335

Merged
merged 2 commits into from
Jun 22, 2022
Merged

Fix a flaky test #3335

merged 2 commits into from
Jun 22, 2022

Conversation

dkhalanskyjb
Copy link
Contributor

Not sure what the test checks for. Does it really need to make all these iterations, or just waiting for every type of exception to fire is enough?

@dkhalanskyjb dkhalanskyjb changed the base branch from master to develop June 21, 2022 08:55
@dkhalanskyjb dkhalanskyjb changed the base branch from develop to master June 21, 2022 08:56

repeat(iterations) {
while (eCnt == 0 || e1Cnt == 0 || e2Cnt == 0 || loopsToMake-- > 0) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The rationale behind this test was the following: given enough parallelism (3 threads) and iterations, we expect to observe every exception type at least once.

Taking into account recent changes in our CI and the nature of this test, I would suggest changing it to be more robust, but keep the original intent: iterate until all exception types are observed OR really many iterations passed (to address flakyness), but add a large enough timeout so CI won't hang for four hours

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given the rationale, I don't understand the need to count iterations, as opposed to just doing this, with something like withTimeout wrapping everything:

Suggested change
while (eCnt == 0 || e1Cnt == 0 || e2Cnt == 0 || loopsToMake-- > 0) {
while (eCnt == 0 || e1Cnt == 0 || e2Cnt == 0) {

Copy link
Contributor

@qwwdfsad qwwdfsad Jun 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking about iterations for fast debugging locally and a large enough timeout for slow CI, but this should work as well, up to you

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think not having iterations, in this case, is more semantically clear. Most other stress tests perform many iterations so that a problematic condition has more chances to arise. Here, it's not the case: we want to wait for a success condition. Waiting for a given number of iterations obscures this, whereas the concept of a timeout is clear and self-evident.

@dkhalanskyjb dkhalanskyjb merged commit 11daf82 into master Jun 22, 2022
@dkhalanskyjb dkhalanskyjb deleted the fix-flaky-test branch June 22, 2022 08:29
mpetrov pushed a commit to mpetrov/kotlinx.coroutines that referenced this pull request Jun 23, 2022
pablobaxter pushed a commit to pablobaxter/kotlinx.coroutines that referenced this pull request Sep 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants