Flutter: getAuthHeader() returns stale cached token when Firebase refresh fails

[beastoin]

Summary

When getIdToken() returns null (Firebase refresh fails), getAuthHeader() falls through to the stale cached token in SharedPreferences instead of clearing it. This causes infinite retry loops with an invalid token.

Root Cause

app/lib/backend/http/shared.dart:33-57 — exact flow:

1. Firebase rotates signing key (e.g., 2c27aff5 rotated Jun 2-3)
2. User's cached ID token (signed with old key) is now invalid
3. App reconnects → getAuthHeader() detects token expired → calls getIdToken()
4. getIdToken() returns null (user's refresh token also expired — requires re-login)
5. refreshedToken is null → SharedPreferences NOT updated → stale token stays in cache
6. hasAuthToken = SharedPreferencesUtil().authToken.isNotEmpty → true (old token still there)
7. !hasAuthToken check at line 49 does NOT trigger → AuthTokenUnavailableException NOT thrown
8. Returns Bearer <stale_expired_token>
9. Backend validates → "Certificate not found" / "Token expired" → close 1008 → retry → same stale token → infinite loop

The null check at line 49 only guards against EMPTY tokens, not STALE tokens.

Runtime Evidence

• 14 users stuck in retry loop since Jun 3 (Firebase key rotation)
• Stale tokens signed with rotated key 2c27aff5 (no longer in Google public key list)
• Tokens expired 3–44 hours, retrying every 15 seconds
• ~5K wasted WebSocket attempts/day from these 14 users
• Zero errors before Jun 3 — started exactly when key rotated

Proposed Fix

When refresh was attempted AND failed, clear the stale cached token:

if (!hasAuthToken || !isExpirationDateValid) {
    final refreshedToken = await AuthService.instance.getIdToken();
    if (refreshedToken != null) {
        SharedPreferencesUtil().authToken = refreshedToken;
    } else if (!isExpirationDateValid) {
        // Refresh failed AND token is known expired — don't use stale token
        SharedPreferencesUtil().authToken = '';
    }
    hasAuthToken = SharedPreferencesUtil().authToken.isNotEmpty;
}

This makes the null-token path reach AuthTokenUnavailableException at line 49-54, which buildHeaders() catches. The existing 401→signOut path then activates, forcing re-login.

Files

• app/lib/backend/http/shared.dart:33-57 — getAuthHeader()
• app/lib/backend/preferences.dart — SharedPreferencesUtil().authToken

Investigation by @mon (runtime signals) and @taro (code tracing) for @beastoin

---

Related Issues & Incident Data

• Incident data: #7631 comment — Firebase key rotation Jun 2-3, 14 affected users, ~5K errors/day
• Sync health investigation: Sync health: 429 retry storm, WAL crashes, v2 token expiry, v1 audio latency #7587 — 6 confirmed sync issues (429 storms, WAL crashes, token expiry, v1 latency)
• Related fix — exponential backoff: Flutter: WebSocket and HTTP retry needs exponential backoff on auth/rate-limit failures #7632 (defense-in-depth, stops retry storm even if stale token persists)
• Related fix — close code differentiation: Backend: differentiate WebSocket auth close codes (expired vs invalid vs re-login) #7633 (enables smart client recovery per auth failure type)

[beastoin]

Incident Data — Firebase Key Rotation (Jun 2-3, 2026)

Trigger Event

Firebase rotated signing key 2c27aff5 around Jun 2-3. This key is NO LONGER in Google's public key list (only 7994b4f3 and 854aa4c2 remain). All ID tokens signed with 2c27aff5 became instantly invalid.

Affected Users

14 users with stale Firebase tokens — either signed with the rotated key ("Certificate not found") or expired 3-44 hours ago ("Token expired"). Zero errors before Jun 3.

Error Volume

• ~5K auth errors/day on backend-listen from these 14 users
• Each user retries WebSocket connection every 15 seconds with the same stale token
• Retry storm triggers Cloud Armor 429 rate limiting on backend-listen

Backend-Listen Error Signals

• All errors from backend-listen service on GKE
• Error types: "Certificate not found" (token signed with rotated key) and "Token expired" (3-44 hours past expiry)
• All return WebSocket close code 1008
• Started exactly Jun 3 — correlates with key rotation timing

Client Behavior Observed

• Users retry every ~2-3 seconds (some faster than the 15s timer due to multiple socket instances)
• Same stale token reused on every retry — getIdToken() returns null but stale cached token is not cleared
• No exponential backoff — fixed interval retries indefinitely
• No max retry count — users have been retrying for 24+ hours

Downstream Impact

• Backend-listen auth validation CPU wasted on ~5K invalid token verifications/day
• Cloud Armor 429s triggered by retry volume from these 14 users
• Masks real auth issues in monitoring — hard to distinguish stale-token retries from genuine auth failures

Incident data from @mon (DevOps), code trace by @taro (Backend Dev) for @beastoin

[beastoin]

Root Cause Found: Regression in commit 9d33d2e (Mar 23, 2026)

The Change That Introduced the Bug

Commit 9d33d2e2d — "fix(app): re-read hasAuthToken and preserve token on null refresh" (Mar 23)

Before (worked correctly for key rotation):

if (!hasAuthToken || !isExpirationDateValid) {
    SharedPreferencesUtil().authToken = await AuthService.instance.getIdToken() ?? '';
}

When getIdToken() returned null → token set to '' → hasAuthToken became false → downstream sign-out path activated. Stale tokens were always cleared.

After (current code, broken):

if (!hasAuthToken || !isExpirationDateValid) {
    final refreshedToken = await AuthService.instance.getIdToken();
    if (refreshedToken != null) {
        SharedPreferencesUtil().authToken = refreshedToken;
    }
    hasAuthToken = SharedPreferencesUtil().authToken.isNotEmpty;
}

When getIdToken() returns null → token is NOT cleared → stale token stays in SharedPreferences → returned as Bearer <stale_token> → infinite retry loop.

Why the Change Was Made (and why it was partially correct)

The commit message says: "preserving near-expiry-but-still-valid tokens on transient refresh failures"

The intent was sound — if a token is about to expire (within 5-min buffer) but still technically valid, and getIdToken() fails transiently (e.g., brief network blip), the old behavior of clearing it to '' would unnecessarily force sign-out. The fix preserved the still-usable token.

The gap: it doesn't distinguish "token still valid but refresh failed" from "token already expired/invalid and refresh also failed." In the second case, preserving the stale token creates the infinite retry loop.

Timeline of Changes

Date	Commit	Change	Effect
Mar 23	9d33d2e2d	Preserve token on null refresh	Introduced stale-token-on-null-refresh behavior
Apr 19	2c714a451	Add AuthTokenUnavailableException + guard in buildHeaders() (#6142)	Added typed exception + catch in buildHeaders. But the guard only fires when token is empty, not stale — so the 9d33 regression was not caught
Apr 20	899a9f6d4	Cherry-pick of above guard (#6848)	Same fix merged to another branch

The Apr 19 guard fix (#6142) addressed a related crash (~3,400 Crashlytics fatals, 83 users) where the token was genuinely empty. But it didn't address the 9d33 regression because in that case the token is non-empty (stale but present) — the !hasAuthToken check at line 49 never fires.

Correct Fix

The proposed fix from the issue body is correct — add the isExpirationDateValid condition:

if (refreshedToken != null) {
    SharedPreferencesUtil().authToken = refreshedToken;
} else if (!isExpirationDateValid) {
    // Refresh failed AND token is known expired — don't use stale token
    SharedPreferencesUtil().authToken = '';
}

This preserves the Mar 23 intent (keep near-expiry-but-valid tokens on transient failure) while fixing the gap (clear tokens that are already known to be expired/invalid).

Regression trace by @taro for @beastoin