Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lease: retry errors claiming lease in LeaseManager::spawn #148

Merged
merged 4 commits into from
Apr 14, 2023

Conversation

hawkw
Copy link
Collaborator

@hawkw hawkw commented Apr 13, 2023

Currently, if the ensure_claimed call in the LeaseManager::spawn
background task fails, the entire task will bail and terminate
immediately. This is not desirable, as the intended use of
LeaseManager::spawn is to spawn a single background task that will
manage leasses for the entire lifetime of a controller process, and it
should not terminate in the face of a transient error claiming the
lease.

This branch changes LeaseManager::spawn to retry transient errors with
an exponential backoff strategy. Exponential backoffs are implemented
using the backoff crate, but --- because this dependency is not
exposed in the public API -- this is strictly an implementation detail,
and the dependency can be swapped out for another exponential backoff
implementation later, if desired. Errors marked as transient and retried
include network errors communicating with the API server, errors parsing
the API server response, and HTTP error statuses returned by the API
server. Errors that result from a bad configuration or missing
permissions are treated as fatal, as they are unlikely to ever succeed
after retrying.

Closes #137
Closes #136

kubert/src/lease.rs Outdated Show resolved Hide resolved
kubert/src/lease.rs Outdated Show resolved Hide resolved
kubert/src/lease.rs Outdated Show resolved Hide resolved
@hawkw hawkw requested a review from adleong April 14, 2023 16:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants