Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add exponential retries on merge errors #480

Merged
merged 23 commits into from
Aug 19, 2020

Conversation

ricardoatsouza
Copy link
Contributor

@ricardoatsouza ricardoatsouza commented Aug 4, 2020

This Pull Request fulfills the following requirements:

  • The commit message follows our guidelines.
  • Tests for the changes have been added if needed.
  • Does not affect documentation or it has been added or updated.

Resolves

Add an exponential retry on merge error, basically retrying up to three times (1 second after the first failure, then 4 seconds later and, finally, again 9 seconds later). If all three fail, an error is thrown.

@ricardoatsouza ricardoatsouza requested review from JrSchild, viestat and a team August 4, 2020 09:32
@github-actions
Copy link

github-actions bot commented Aug 4, 2020

Pull Request Test Coverage Report for Build 194556948

  • 25 of 25 (100.0%) changed or added relevant lines in 4 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage increased (+1.3%) to 96.313%

Totals Coverage Status
Change from base Build 189737946: 1.3%
Covered Lines: 141
Relevant Lines: 145

💛 - Coveralls

@viestat
Copy link
Member

viestat commented Aug 4, 2020

Why is the distfolder removed? As far as I know, GitHub Actions do not work with TypeScript. Did something changed?

@ricardoatsouza ricardoatsouza force-pushed the feat/CP-2089_retry_PR_on_merge_errors branch from 16c3132 to 9289fbb Compare August 4, 2020 10:15
Copy link
Member

@viestat viestat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good the only thing I am missing is the README changes, there we can describe how to configure the number of retires.

@ricardoatsouza
Copy link
Contributor Author

Looks good the only thing I am missing is the README changes, there we can describe how to configure the number of retires.

README.MD updated in c97bfc0.

README.md Show resolved Hide resolved
src/common/merge.ts Outdated Show resolved Hide resolved
src/common/merge.ts Outdated Show resolved Hide resolved
src/common/merge.ts Outdated Show resolved Hide resolved
src/eventHandlers/checkSuite/index.spec.ts Show resolved Hide resolved
src/eventHandlers/checkSuite/index.spec.ts Outdated Show resolved Hide resolved
src/eventHandlers/checkSuite/index.spec.ts Show resolved Hide resolved
Copy link
Contributor

@JrSchild JrSchild left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See comment

Copy link
Contributor

@merlinnot merlinnot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only some errors should be retried, I don't think we should retry in all cases. As an example, the "out of sync" error is not retriable, as the branch must be rebased. Some other errors, such as the one originally reported, are cause by GitHub not being done processing yet. This should be retried.

Copy link
Contributor

@JrSchild JrSchild left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One small comment. This is indeed better with .times(3). Also see Natan's review.

src/eventHandlers/checkSuite/index.spec.ts Outdated Show resolved Hide resolved
Copy link
Contributor

@JrSchild JrSchild left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also we should overwrite Date.now or something similar to speed up the tests. We should never actually be backing off that long during the test.

…dedott/merge-me-action into feat/CP-2089_retry_PR_on_merge_errors
@ricardoatsouza
Copy link
Contributor Author

@JrSchild @merlinnot

Added parameter to speed up tests: ef0cedc

Retrying only on the reported error: 7cf6771 ( appreciate feedback on this one)

Copy link
Contributor

@JrSchild JrSchild left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ricardoatsouza can you please revert #ef0cedc3674f849c0f0120c0d0d990d2f28537f1. You are exposing an internal implementation by changing the interface, but the option is never used in production. Also starting the backoff from 100ms instead is definitelly not the correct way to "speed up the test". Any timers that run longer than the an immediate interval should be avoided as much as possible. See our (internal) firestore-extensions package (retries.spec.ts) on how to properly test it.

Can you add a test that makes sure backoff does not happen if shouldRetry does not match the error?

src/eventHandlers/checkSuite/index.spec.ts Outdated Show resolved Hide resolved
@ricardoatsouza
Copy link
Contributor Author

@ricardoatsouza can you please revert #ef0cedc3674f849c0f0120c0d0d990d2f28537f1. You are exposing an internal implementation by changing the interface, but the option is never used in production. Also starting the backoff from 100ms instead is definitelly not the correct way to "speed up the test". Any timers that run longer than the an immediate interval should be avoided as much as possible. See our (internal) firestore-extensions package (retries.spec.ts) on how to properly test it.

It's not the wrong way either. Regardless:

Can you add a test that makes sure backoff does not happen if shouldRetry does not match the error?

Added in d5400ab

@ricardoatsouza ricardoatsouza merged commit 7f3d613 into master Aug 19, 2020
@ricardoatsouza ricardoatsouza deleted the feat/CP-2089_retry_PR_on_merge_errors branch August 19, 2020 07:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants