New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Tweak BuildPulse/rspec-retry
logic
#11598
Conversation
Review period ended. |
89b9ab1
to
0bc0d4c
Compare
0bc0d4c
to
06c4e3c
Compare
06c4e3c
to
87782eb
Compare
87782eb
to
19e0592
Compare
# Retry multiple times when using BuildPulse to detect and submit | ||
# flakiness (because rspec-retry is disabled). | ||
if [ -n "$HOMEBREW_BUILDPULSE_ACCESS_KEY_ID" ]; then | ||
brew tests --online --coverage || \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A quick question to check my understanding of things: Am I right in thinking that each call to brew tests
will run the Homebrew.tests
method?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jasonrudolph Yup!
01d0474
19e0592
to
01d0474
Compare
01d0474
to
d42aff2
Compare
BuildPulse is trying to find flaky tests for us but, given the previous model of using `rspec-retry`, it would rarely find them. Instead, let's try to always rerun `brew tests` multiple times, report to BuildPulse each time (by moving the reporting logic into `brew tests`) and disable `rspec-retry` when using BuildPulse. While we're here, let's enable `rspec-retry` locally so we don't have flaky tests biting maintainers/contributors there.
d42aff2
to
a4c2e0e
Compare
BuildPulse is trying to find flaky tests for us but, given the previous model of using
rspec-retry
, it would rarely find them. Instead, let's try to always rerunbrew tests
multiple times, report to BuildPulse each time (by moving the reporting logic intobrew tests
) and disablerspec-retry
when using BuildPulse.While we're here, let's enable
rspec-retry
locally so we don't have flaky tests biting maintainers/contributors there.CC @jasonrudolph for thoughts