SSO token forwarding fails after idle period: stale ID token with zero ExpiresAt never evicted from proxy store

[gusevda]

Summary

After an idle period of >30 minutes with no incoming requests, muster forwards expired JWT ID tokens to downstream forwardToken: true MCP servers, resulting in 401 Unauthorized errors. Token refresh from the client (Backstage) succeeds at the muster level, but the stale ID token continues to be forwarded.

Root Cause

storeIDTokenForSSO() in internal/aggregator/auth_resource.go:226 stores tokens with only the IDToken field populated — no AccessToken, no ExpiresIn, no ExpiresAt:

oh.StoreToken(familyID, userID, musterIssuer, &api.OAuthToken{IDToken: idToken})

When TokenStore.Store() calls SetExpiresAtFromExpiresIn(), it's a no-op (ExpiresIn is 0), leaving ExpiresAt as zero. IsExpiredWithMargin() in pkg/oauth/types.go:98 treats zero as "never expires":

if t.ExpiresAt.IsZero() {
    return false // Tokens without expiration don't expire
}

This means GetByIssuer() returns these stale tokens indefinitely, regardless of the embedded JWT's actual exp claim.

Mechanism

1. User logs in → SessionCreationHandler fires → storeIDTokenForSSO stores ID-only token with zero ExpiresAt
2. Proactive refresh (triggered by incoming requests) keeps the upstream token fresh and calls TokenRefreshHandler → storeIDTokenForSSO updates the proxy store
3. After >30 min idle, the upstream Dex token expires and no proactive refresh runs (it only triggers on incoming requests)
4. Next request arrives → client-side refresh succeeds (muster rotates its own tokens), but the upstream Dex token is NOT refreshed → TokenRefreshHandler never fires
5. getIDTokenForForwarding() reads from the proxy store → gets the stale zero-ExpiresAt token with expired JWT inside
6. Downstream MCP server validates the JWT exp claim → rejects with 401 invalid_token

Additional Impact

Background SSE listener connections (listen to server forever) enter an infinite 1-second retry loop with stale tokens, causing:

• Persistent log flooding: ERROR: failed to listen to server. retry in 1 second: 401 invalid_token
• Unnecessary load on downstream servers

Reproduction

Unit test (instant)

Branch test/stale-id-token-forwarding contains TestTokenStore_IDOnlyTokenWithExpiredJWT_NeverEvicted in internal/oauth/token_store_test.go that demonstrates the core store behavior.

go test ./internal/oauth/ -run TestTokenStore_IDOnlyTokenWithExpiredJWT_NeverEvicted -v

End-to-end (35 minutes)

1. Login from Backstage to muster
2. Make a request targeting a forwardToken: true server (e.g., gazelle-mcp-kubernetes) — succeeds
3. Wait >30 minutes without making any requests
4. Make the same request — fails with 401 Unauthorized

Verified on

• Local dev instance (token forwarding to graveler-mcp-kubernetes)
• Deployed instance on gazelle cluster (token forwarding to gazelle-mcp-kubernetes)
  • Login at 06:58 UTC, successful call at 08:28 UTC, failed call at 09:51 UTC after ~80 min idle gap
  • Log: CallTool failed for x_gazelle-mcp-kubernetes_capi_list_clusters: failed to initialize on-demand client for gazelle-mcp-kubernetes: authentication required: server returned 401 Unauthorized

Key Files

File	Relevance
internal/aggregator/auth_resource.go:226-237	storeIDTokenForSSO stores ID-only token with no expiry
pkg/oauth/types.go:96-102	IsExpiredWithMargin returns false for zero ExpiresAt
internal/oauth/token_store.go:130-146	GetByIssuer returns stale zero-expiry tokens
internal/aggregator/connection_helper.go:242-262	getIDTokenForForwarding reads from proxy store
internal/aggregator/connection_helper.go:890-946	makeTokenForwardingHeaderFunc forwards stale token
internal/aggregator/server.go:1419-1433	TokenRefreshHandler only fires on proactive refresh

Possible Fixes

(a) Set ExpiresAt on ID-only tokens based on the JWT exp claim in storeIDTokenForSSO

(b) Trigger an upstream Dex refresh when the client refreshes and the stored upstream token is expired

(c) Check the JWT exp claim in getIDTokenForForwarding before forwarding, and trigger re-authentication if expired

[gusevda]

Why first attempt to reproduce the issue by waiting 35 minutes without requests failed? Only after longer period it stopped working?

The timeline:

• 06:58 — Login
• 08:28 (~90 min idle) — Tool call succeeded
• 09:51 (~83 min idle from 08:28) — Tool call failed

The 35-min wait at 08:28 should have broken the proactive refresh chain, yet it worked. Here's what I think happened:

At 08:28 (success): When Backstage refreshed the muster token, RefreshAccessToken in mcp-oauth called provider.RefreshToken() using the upstream Dex refresh token stored from the original login. This refresh token was still valid (Dex refresh tokens have a long lifetime). Dex returned a fresh ID token, fireTokenRefreshHandler fired, the proxy store got updated, AND the onAuthenticated middleware injected the fresh token into the request context. Tool call succeeded.

At 09:51 (failure): The same flow ran, but now the provider token mapping was different. After the 08:28 refresh, the new provider token was saved with Dex's native 30-min expiry (not the 24h extended expiry from the initial login). By 09:51, this provider token was 83 minutes expired. If the mcp-oauth store cleaned it up or if the Dex refresh token from 08:28 was rotated/invalidated, getProviderToken() in the middleware would return nil — no ID token injected into context, fallback to the stale proxy store entry → 401.

The key insight: the first idle gap is survivable because the initial login's extended 24h provider token and fresh Dex refresh token are still available. But subsequent idle gaps after a refresh are more fragile because the provider token loses its 24h extension — extendTokenExpiryForStorage only runs during the initial code exchange, not during refreshes.

[QuentinBisson]

Filed giantswarm/mcp-oauth#285 for the on-demand refresh path (option B from #549).

Once that lands, the muster-side follow-up wires mcpoauth.Server.RefreshSession(ctx, familyID) into getIDTokenForForwarding: if the cached token is expired, refresh in-process and re-read before forwarding.

The current PR (#620) closes the storage-side leak (option A) only — ExpiresAt is now correctly set from the JWT exp, so stale tokens are evictable. The on-demand recovery is the second half and depends on mcp-oauth#285.