fix(control-plane): revoke orphaned secrets when createContainer fails after injectSecrets - #8255
Merged
Merged
Conversation
…s after injectSecrets The #8202 reorder (database -> injectSecrets -> createContainer) opened a window where injectSecrets succeeds and mints secretRef, then createContainer fails right after -- since provisionTenant always rethrows rather than returning on a step failure, secretRef never reached the caller to persist and later revoke, permanently orphaning a live credential in the broker. Unreachable before the reorder, since injectSecrets used to be the last step. provisionTenant now best-effort revokes that secretRef itself before rethrowing, and the PagerDuty alert carries it too as a fallback for when the revoke itself fails (e.g. broker unreachable).
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8255 +/- ##
==========================================
- Coverage 92.13% 91.60% -0.53%
==========================================
Files 783 783
Lines 78605 78645 +40
Branches 23733 23734 +1
==========================================
- Hits 72421 72046 -375
- Misses 5062 5524 +462
+ Partials 1122 1075 -47
Flags with carried forward coverage won't be shown. Click here to find out more.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes a real regression in #8253 (already merged), caught by code review after merge.
#8253 reordered
provisionTenanttodatabase -> injectSecrets -> createContainerso a tenant's bootstrap secret exists before the container's one real cold-bootstub.start()call. That reorder opened a window that didn't exist before:injectSecretscan succeed (custodying a real secret + mintingsecretRef) and thencreateContainercan still fail right after it -- a real path (Cloudflare quota, a transient container-API error). SinceprovisionTenantalways rethrows rather than returning on a step failure,secretRefnever reached the caller to persist and later revoke viadeprovisionTenant-- permanently orphaning a live, exchangeable credential in the broker with no code path left to clean it up.This was unreachable before #8253:
injectSecretsused to be the last of the three steps, so nothing after it could ever fail oncesecretRefwas set.Fix
provisionTenant's catch block now best-effort revokes the just-injected secret (when one was actually obtained) before rethrowing -- self-contained cleanup, doesn't depend on any caller doing the right thing with a value it was never going to receive.onFailurehook -- it must never mask the real provisioning error.secretRefis also threaded into the PagerDuty alert'scustomDetails(bothprovisionTenantanddeprovisionTenant) as defense-in-depth for when the best-effort revoke itself fails -- an operator paged for the failure has something to manually revoke by instead of nothing.Tests
secretRef, and the originalcreateContainererror is still what's rethrown.secretRefwas ever obtained (provisionDatabaseorinjectSecretsitself failing) -- confirms the fix is correctly scoped, not a blanket "always try to revoke."secretRefwhen present, omits it when absent (existing exact-equality test already covered the omission case).deprovisionTenant's alert also carries thesecretRefit was given.Verified: control-plane build + full
node:testsuite (201/201) + real c8/lcov coverage (100% line+branch on both touched files),git diff --checkclean.Test plan
control-plane:npm run build && npm run test:node(201/201)control-plane: real coverage vianpm run control-plane:coverage-- 100% line+branch onprovisioning.tsandpagerduty-notify.tsgit diff --check