Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reintroduce renew lease optimization #10462

Merged
merged 10 commits into from
Aug 30, 2022

Conversation

t0yv0
Copy link
Member

@t0yv0 t0yv0 commented Aug 22, 2022

Description

Re-introduce the optimization that avoids calling renewLease on critical startup path. Previous attempt caused issues and was reverted (see Postmortem). The underlying backend issue with a short expiration on initial token has been patched, so this is safe to reintroduce. Besides reintroducing the original code, the PR addresses @RobbieMcKinstry concern that we cannot solely rely on background ticker to keep the token fresh (up-to-date). To address this, we chose to block the caller of GetToken and refresh the token right then rather than risking returning a stale token.

Fixes https://github.com/pulumi/home/issues/2200

Checklist

  • I have added tests that prove my fix is effective or that my feature works
  • Yes, there are changes in this PR that warrants bumping the Pulumi Service API version

@t0yv0
Copy link
Member Author

t0yv0 commented Aug 22, 2022

@RobbieMcKinstry WDYT? I'm considering maybe lifting the refresh logic out to spell out some unit tests as this is getting more and more tricky.

@t0yv0
Copy link
Member Author

t0yv0 commented Aug 22, 2022

Ah looks like our integration tests are flagging this as invalid, working it out.

Copy link
Contributor

@RobbieMcKinstry RobbieMcKinstry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good overall. I feel less confident with my review after the last time however ;D

pkg/backend/httpstate/state.go Outdated Show resolved Hide resolved
@t0yv0 t0yv0 force-pushed the t0yv0/reintroduce-renew-lease-optimization branch from dd2f1d2 to 397bfbd Compare August 29, 2022 18:52
// If ticker has not kept up, block on
// renewing rather than risking returning a
// stale token.
renewUpdateLeaseIfStale()
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oof - i've tried adding a test for this case, but it seems like it's not possible to reach here. The problem is that we need an active token to renew a lease. If we're detecting we've not kept up with refreshing tokens, we have a stale token, then we won't be able to review it either. There is some grace here perhaps that the staleness check indicates a need to renew slightly before actual expiration. So if the ticker fails completely, this safeguard might be beneficial to keep. How to test this though..

@t0yv0 t0yv0 marked this pull request as ready for review August 29, 2022 22:04
@RobbieMcKinstry
Copy link
Contributor

@t0yv0 looks good to me!

@t0yv0 t0yv0 merged commit 70f7677 into master Aug 30, 2022
@pulumi-bot pulumi-bot deleted the t0yv0/reintroduce-renew-lease-optimization branch August 30, 2022 18:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants