Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(Should fix) #3789: Fix flaky test retry mechanism for Bazel CI #3908

Merged
merged 5 commits into from
Oct 7, 2021

Conversation

BenHenning
Copy link
Sponsor Member

@BenHenning BenHenning commented Oct 6, 2021

Explanation

(Should fix) #3789

See #3789 (comment) for context on the root issue. This PR fixes the retry mechanism by temporarily disabling early exit when performing the Bazel buld retry, and then re-enables it & conditionally exits with error code if the last command fails (since Bazel should guarantee that if any of the iterations result in a passing build, all subsequent ones should).

Note that actually fixing this does mean that legitimate build failures will result in spammier output since they'll be retried multiple times, but that should probably be fine until we can fix the Jettifier issue long-term (#3759).

Finally, I can't actually seem to verify that this is working until the PR is merged (I tried multiple times in #3790 to trigger a scenario in the PR, but couldn't). I think it's reasonable to merge this as-is, and then close #3789 once it's been verified (I'm intentionally not linking the PR & issue for this reason since we can't be sure it's fixed until a few days after merging).

For partial verification:

Essential Checklist

  • The PR title and explanation each start with "Fix #bugnum: " (If this PR fixes part of an issue, prefix the title with "Fix part of #bugnum: ...".)
  • Any changes to scripts/assets files have their rationale included in the PR explanation.
  • The PR follows the style guide.
  • The PR does not contain any unnecessary code changes from Android Studio (reference).
  • The PR is made from a branch that's not called "develop" and is up-to-date with "develop".
  • The PR is assigned to the appropriate reviewers (reference).

For UI-specific PRs only

N/A

@BenHenning BenHenning changed the title Fix #3789: Fix flaky test retry mechanism for Bazel CI [RunAllTests] Fix #3789: Fix flaky test retry mechanism for Bazel CI Oct 6, 2021
@BenHenning BenHenning closed this Oct 6, 2021
@BenHenning
Copy link
Sponsor Member Author

Reopening to trigger run-all-tests workflow.

@BenHenning BenHenning reopened this Oct 6, 2021
@BenHenning BenHenning changed the title [RunAllTests] Fix #3789: Fix flaky test retry mechanism for Bazel CI [RunAllTests] Should fix #3789: Fix flaky test retry mechanism for Bazel CI Oct 6, 2021
@BenHenning BenHenning linked an issue Oct 6, 2021 that may be closed by this pull request
@BenHenning BenHenning changed the title [RunAllTests] Should fix #3789: Fix flaky test retry mechanism for Bazel CI [RunAllTests] Fix flaky test retry mechanism for Bazel CI Oct 6, 2021
@BenHenning
Copy link
Sponsor Member Author

Follow-up: I can't seem to unlink #3789 so I'll need to manually reopen it.

@BenHenning BenHenning changed the title [RunAllTests] Fix flaky test retry mechanism for Bazel CI [RunAllTests] (Should fix) #3789: Fix flaky test retry mechanism for Bazel CI Oct 6, 2021
@BenHenning BenHenning changed the title [RunAllTests] (Should fix) #3789: Fix flaky test retry mechanism for Bazel CI (Should fix) #3789: Fix flaky test retry mechanism for Bazel CI Oct 6, 2021
@BenHenning BenHenning marked this pull request as ready for review October 6, 2021 22:16
@BenHenning
Copy link
Sponsor Member Author

BenHenning commented Oct 6, 2021

TODO script failure is probably due to an out-of-date base branch. Syncing to see if CI now passes.

Edit: it seems to be passing now.

Copy link
Contributor

@anandwana001 anandwana001 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As this can be verified after few days of merging so this LGTM and we can merge it.

@oppiabot oppiabot bot added the PR: LGTM label Oct 7, 2021
@oppiabot
Copy link

oppiabot bot commented Oct 7, 2021

Hi @BenHenning, this PR is ready to be merged. Please address any remaining comments prior to merging, and feel free to merge this PR once the CI checks pass and you're happy with it. Thanks!

@BenHenning
Copy link
Sponsor Member Author

Thanks @anandwana001. Merging now.

@BenHenning BenHenning merged commit dd4cdca into develop Oct 7, 2021
@BenHenning BenHenning deleted the fix-bazel-build-retry-mechanism branch October 7, 2021 06:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Sharded Bazel tests are build-flaky in CI
2 participants