cli: retry auth handshake deadline exceeded errors in CLI and Terraform #2976
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Context
CLI (and also our Terraform provider) uses gRPC to talk with different services on Constellation.
We have a retry logic for these calls that make sure transient errors don't cause the entire action to fail.
In the past we have observed "authentication handshake failed: context deadline exceeded" during these rpcs.
Since we do not set a deadline for some of our CLI commands (e.g.
constellation recover
) but we have observed these errors nonetheless, I believe that gRPC/Go TLS uses a custom deadline at some point during the handshake.This deadline may be exceeded by transient connection errors and should be retried by our code.
Proposed change(s)
authentication handshake failed: context deadline exceeded
to the list of retriable rpc errorsRelated issue
This PR may resolve the following issues, though we should keep them open and observe potential regressions:
Checklist