Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cli: retry auth handshake deadline exceeded errors in CLI and Terraform #2976

Merged
merged 1 commit into from
Mar 8, 2024

Conversation

daniel-weisse
Copy link
Member

@daniel-weisse daniel-weisse commented Mar 8, 2024

Context

CLI (and also our Terraform provider) uses gRPC to talk with different services on Constellation.
We have a retry logic for these calls that make sure transient errors don't cause the entire action to fail.
In the past we have observed "authentication handshake failed: context deadline exceeded" during these rpcs.

Since we do not set a deadline for some of our CLI commands (e.g. constellation recover) but we have observed these errors nonetheless, I believe that gRPC/Go TLS uses a custom deadline at some point during the handshake.
This deadline may be exceeded by transient connection errors and should be retried by our code.

Proposed change(s)

  • add authentication handshake failed: context deadline exceeded to the list of retriable rpc errors

Related issue

This PR may resolve the following issues, though we should keep them open and observe potential regressions:

Checklist

  • Run the E2E tests that are relevant to this PR's changes
  • Update docs
  • Add labels (e.g., for changelog category)
  • Is PR title adequate for changelog?
  • Link to Milestone

Signed-off-by: Daniel Weiße <dw@edgeless.systems>
@daniel-weisse daniel-weisse added the bug fix Fixing a bug label Mar 8, 2024
@daniel-weisse daniel-weisse added this to the v2.17.0 milestone Mar 8, 2024
Copy link

netlify bot commented Mar 8, 2024

Deploy Preview for constellation-docs ready!

Name Link
🔨 Latest commit 75b2a3a
🔍 Latest deploy log https://app.netlify.com/sites/constellation-docs/deploys/65eae79a532b500008f8cb63
😎 Deploy Preview https://deploy-preview-2976--constellation-docs.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

Copy link
Contributor

github-actions bot commented Mar 8, 2024

Coverage report

Package Old New Trend
internal/grpc/retry 90.90% 92.30% ↗️

Copy link
Contributor

@burgerdev burgerdev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, thanks!

@daniel-weisse daniel-weisse merged commit 2733049 into main Mar 8, 2024
11 checks passed
@daniel-weisse daniel-weisse deleted the fix/cli/handshake-context-deadline-exceeded branch March 8, 2024 12:15
malt3 pushed a commit that referenced this pull request Mar 11, 2024
…rm (#2976)

Signed-off-by: Daniel Weiße <dw@edgeless.systems>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug fix Fixing a bug
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants