Skip to content

fix(oauth): persist reconnect ID token to stop SSO re-exchange rotation storm#981

Closed
paurosello wants to merge 2 commits into
mainfrom
worktree-fix-garm-reexchange-idtoken-persist
Closed

fix(oauth): persist reconnect ID token to stop SSO re-exchange rotation storm#981
paurosello wants to merge 2 commits into
mainfrom
worktree-fix-garm-reexchange-idtoken-persist

Conversation

@paurosello

Copy link
Copy Markdown
Contributor

Problem

A long-lived SSO session (human dex login) got deauthed on a management cluster despite the mcp-oauth rotation-race work. Root cause is a muster lifecycle gap, independent of the mcp-oauth provider-token rotation race (giantswarm/giantswarm#37164):

  1. A session reconnects after its login-time ID token expired (e.g. after a pod restart). onAuthenticated takes the authAlive branch and re-inits SSO via initSSOForSession using the ID token from the live request context — but never persists it to the OAuth-proxy store.
  2. The background re-exchange/forwarding closure getIDTokenForForwarding runs on a detached context.Background() and can only read that store, so it finds nothing → logs no subject ID token available for re-exchange → falls back to the in-process refresher RefreshSessionRefreshAccessToken, which rotates the client's mcp refresh token.
  3. On a token-exchange backend whose continuous-listen retries every ~1s, this rotated the refresh-token family ~56×/min for ~15 min until two rotations collided → OAuth 2.1 refresh-token reuse detection revoked the whole family (all user+client tokens) → session deauthed.

Connect-time token exchange kept succeeding the whole time (it reads the live request context), which is why the failure looked intermittent.

Fix

Persist the request-context ID token in initSSOForSession (via the existing storeIDTokenForSSO, which already no-ops on empty/unparseable tokens). Because onAuthenticated re-inits on active requests, the proxy store now stays fresh for as long as the session keeps making authenticated requests, so the background re-exchange resolves a subject from the store and never hits the rotating refresher.

Scope / follow-up

This kills the trigger. It does not remove the underlying hazard that the background SSO re-exchange refresher (RefreshSession) rotates the client-facing refresh token at all. A proper follow-up is an mcp-oauth provider-only refresh entry point that fires TokenRefreshHandler (repopulating the ID token) without rotating the client's mcp refresh token; muster's re-exchange path would use that instead. Tracked as a follow-up so this safe, well-scoped fix can land first.

Not fixed by bumping mcp-oauth to v1.1.1 — that coordinates the upstream provider token, not muster's SSO ID-token lifecycle or the client-refresh-token rotation.

Tests

  • New TestInitSSOForSession_PersistsIDToken (+ no-op-on-empty subtest).
  • Full internal/aggregator package passes with -race; go vet clean.

…on storm

A long-lived SSO session that reconnects after its login-time ID token has
expired (e.g. after a pod restart) re-inits its SSO backend connections in
initSSOForSession using the ID token from the live request context, but never
persisted that token to the OAuth-proxy store. The background re-exchange /
forwarding closure (getIDTokenForForwarding) runs on a detached
context.Background() and can only read that store, so it found nothing and
logged "no subject ID token available for re-exchange", then fell back to the
in-process refresher (RefreshSession -> RefreshAccessToken), which rotates the
client's mcp refresh token. On a token-exchange backend whose continuous-listen
retries every ~1s, this rotated the refresh-token family ~56x/min until two
rotations collided and OAuth 2.1 reuse detection revoked the whole family,
deauthing the session.

Persist the request-context ID token in initSSOForSession so the store stays
populated for as long as the session keeps making authenticated requests, which
lets the background re-exchange resolve a subject without triggering the
refresher. storeIDTokenForSSO already no-ops on empty/unparseable tokens.

This is independent of the mcp-oauth provider-token rotation race
(giantswarm/giantswarm#37164); a follow-up should stop the background
re-exchange from rotating the client-facing refresh token at all (an mcp-oauth
provider-only refresh that fires TokenRefreshHandler without rotating).
@paurosello
paurosello marked this pull request as ready for review July 20, 2026 13:07
@paurosello
paurosello requested a review from a team as a code owner July 20, 2026 13:07
@paurosello

Copy link
Copy Markdown
Contributor Author

Superseded — the fix has been folded directly into the rotation-race branch #977 (commit a46fd8e), which also bumps mcp-oauth to the latest v1.1.2. That way graveler gets both the id_token-persistence fix and the rotation-race work in one build. See #977 for the full writeup.

@paurosello paurosello closed this Jul 20, 2026
@paurosello
paurosello deleted the worktree-fix-garm-reexchange-idtoken-persist branch July 20, 2026 13:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant