fix: stop background worker 401 storm when OAuth token expires#463
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Sentry Issue: LOREAI-GATEWAY-Z — 19 users, 2,349 events
When a single user's OAuth bearer token expires, background workers (distillation, curation, consolidation) keep retrying every 30 seconds with the stale token. Each attempt generates a
Sentry.captureExceptioncall, flooding Sentry with thousands of events.Root Cause
PR #454 added a retry-once mechanism: mark the session credential stale → fall back to global → retry if credential changed. But in single-session OAuth setups (the typical Claude Code user), the session and global credentials are the same expired token. So:
markAuthStale(sessionID)marks the session staleresolveAuth(sessionID)skips the stale session, falls through togetLastSeenAuth()credentialChanged = falseSentry.captureException()firesFix (three layers)
auth.ts—resolveAuth()detects same-token fallback: When the stale session credential and global credential have the same value, returnnullinstead of the expired global token. This lets callers know there's no usable credential available.idle.ts— skip background work on stale auth: Before scheduling idle work (distillation, curation, consolidation), checkisAuthStale(sessionID) && !resolveAuth(sessionID). If auth is stale and no fresh credential is available, skip the session entirely. Auth refreshes when the next client request arrives.instrument.ts— filter auth errors inbeforeSend: Add/Worker upstream auth error/toTRANSIENT_ERROR_PATTERNSas defense-in-depth, suppressing any residual auth error events that slip through.Tests
resolveAuthsame-token detection