lease: retry errors claiming lease in LeaseManager::spawn
#148
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, if the
ensure_claimed
call in theLeaseManager::spawn
background task fails, the entire task will bail and terminate
immediately. This is not desirable, as the intended use of
LeaseManager::spawn
is to spawn a single background task that willmanage leasses for the entire lifetime of a controller process, and it
should not terminate in the face of a transient error claiming the
lease.
This branch changes
LeaseManager::spawn
to retry transient errors withan exponential backoff strategy. Exponential backoffs are implemented
using the
backoff
crate, but --- because this dependency is notexposed in the public API -- this is strictly an implementation detail,
and the dependency can be swapped out for another exponential backoff
implementation later, if desired. Errors marked as transient and retried
include network errors communicating with the API server, errors parsing
the API server response, and HTTP error statuses returned by the API
server. Errors that result from a bad configuration or missing
permissions are treated as fatal, as they are unlikely to ever succeed
after retrying.
Closes #137
Closes #136