Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

curtin-ci: retry git clone and fetch a few times #85

Merged
merged 2 commits into from Oct 28, 2019

Conversation

@paride
Copy link
Member

paride commented Oct 24, 2019

This makes the job less prone to fail in case of network hiccups.

This makes the job less prone to fail in case of network hiccups.
curtin/jobs-ci.yaml Outdated Show resolved Hide resolved
git fetch upstream --tags
cmd="git fetch upstream --tags"
for i in 0s 5s 1m 10m; do sleep $i; $cmd && s=0 && break || s=$?; done; (exit $s)

This comment has been minimized.

Copy link
@blackboxsw

blackboxsw Oct 25, 2019

Contributor

Since the retry sleep counts are reused by both git commands, maybe it's worth either a function a variable

RETRY_SLEEPS=(0s 5s 1m 10m)

for i in ${RETRY_SLEEPS[@]}; do sleep $i; $cmd && break || s=$?; done;

This comment has been minimized.

Copy link
@paride

paride Oct 28, 2019

Author Member

Thanks Chad; actually it's the rest of the retry logic that I would like not to repeat. I made it into a function, leaving the retry intervals customizable, but after thinking on it a bit more I'm convinced that we can have some default that will cover almost every case of our interest. So I propose this function:

retry() (for i in {0..5}; do sleep $((8*i**3)) && "${@:1}" && break; done)

which has some nice properties:

  • Retries 5 times over half an hour (that is, has strong defaults)
  • It retries with a cubic backoff strategy, waiting little time after the first failures, increasing the wait time if the command's keeps failing
  • "${@:1}" expands as a bash array, which is way better than using a variable to store the command to run ($cmd in the previous implementation)
  • Preserves the retried command exit code
  • Runs in a subshell, thus not leaking the for index variable
  • It's short and reusable
  • It's shellcheck clean
@paride

This comment has been minimized.

Copy link
Member Author

paride commented Oct 28, 2019

Pushed a new commit with the new retry() function. Once the function is defined to retry a command it's sufficient to add a retry in front of it.

Copy link
Contributor

blackboxsw left a comment

LGTM. Thanks Paride good choice.

@blackboxsw blackboxsw merged commit 7d59593 into CanonicalLtd:master Oct 28, 2019
1 check passed
1 check passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.