Skip to content

Reduce test retry count to zero#9380

Closed
Renkai wants to merge 1 commit intoapache:masterfrom
Renkai:reduce-retry
Closed

Reduce test retry count to zero#9380
Renkai wants to merge 1 commit intoapache:masterfrom
Renkai:reduce-retry

Conversation

@Renkai
Copy link
Contributor

@Renkai Renkai commented Jan 31, 2021

Our current tests will auto-retry failed tests, which could lead to some problems:

  • If one method in one test class failed, all methods in the test class will be retriggered, which will make the test easier to timeout and generate more noise logs to make us harder to locate the error log.
  • The failed method may have changed the test environment, and then the second try may report different error info, which will mislead us to solve the problem, too.
  • Auto retry is useful to pass flaky tests, but have a low return on investment for the same reason shown above.

Signed-off-by: Renkai <gaelookair@gmail.com>
@Renkai
Copy link
Contributor Author

Renkai commented Jan 31, 2021

/pulsarbot run-failure-checks

1 similar comment
@Renkai
Copy link
Contributor Author

Renkai commented Jan 31, 2021

/pulsarbot run-failure-checks

@eolivelli
Copy link
Contributor

Thanks @Renkai

I had already files the same patch at #9043

There is an open discussion.
Can you please add your opinion in that patch?

I am totally +1 on this patch

@Renkai
Copy link
Contributor Author

Renkai commented Feb 1, 2021

@eolivelli Seems set retry count to 0 really have a significant effect on the success rate on our code base, maybe it's not suitable to be merged now, I will leave this patch to see which tests are easy to be flaky

Copy link
Member

@sijie sijie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before merging this, we should have a clear idea about how this PR will impact CI.

@Renkai Renkai closed this Feb 2, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

Comments