Repository navigation
[Bug][v0.10.0] Long-running session stays stuck on a revoked Anthropic OAuth token (401 'has been revoked'); /reload does not recover #3575
Replies: 4 comments
|
We merged #3393 earlier today so /reload rereads the credential store and refreshes the running session's credentials. This will be in the next stable release. Automatic recovery after a revoked-token 401 is still separate, so keeping this open. |
|
This looks like the same root cause as #3669, seen from the expired-token side. In my case a session held by a worker for about three days failed with This doesn't only hit sessions that sit idle and get resumed later. It also hits sessions that are actively working. The session stops mid-task, and its subagents stop with it. That makes Prime Agent hard to rely on for long-running tasks, which is where it should be strongest. |
|
Still reproducing on prime-agent 0.10.1, macOS arm64, using the release binary in daemon mode with Anthropic OAuth and One long-running session has required repeated worker restarts. Latest sequence (UTC):
The latest restart is confirmed; we have not yet verified a successful model call after that restart. Earlier worker restarts restored the session, but the failure returned after later credential expiry/rotation. A local workaround that restarts only idle, unattached workers cannot reliably protect long-running or heartbeat-driven sessions without interrupting work. Automatic credential re-resolution/recovery is needed in the worker request path, rather than requiring periodic process restarts. We have not tested |
|
Also reproducing every day on prime-agent 0.10.0 → 0.10.1, macOS arm64, daemon mode, Anthropic OAuth, What we see (times UTC):
Each time, the only recovery was a human opening the sessions in the UI. A heartbeat-driven session can't recover on its own, because the heartbeat turn itself is the thing that fails. So for anyone using heartbeats for daily work, this bug makes them unusable without a person checking in every morning. A side effect, in case it helps: after the 26 Oct failure, the queued heartbeat prompts for Saturday, Sunday and Monday were all delivered together when the session came back, about 20 minutes later. |
Uh oh!
There was an error while loading. Please reload this page.
Summary
A long-running session worker keeps sending an Anthropic OAuth access token that has been revoked. It never recovers. Other sessions on the same machine refresh the shared token in
auth.json, and the old worker keeps using its own copy. Every later turn fails with:This hits sessions driven by heartbeats hardest. They live for days, so each heartbeat run fails the same way.
Observed (v0.10.0, Linux x64, binary install, daemon mode, Anthropic OAuth / Claude Max)
auth.json.401 OAuth access token has expiredat 23:13Z. That was one minute before a new session refreshed the token.401 ... has been revoked: 14 heartbeat runs over about 10.5 hours, plus manual prompts. Other sessions used the fresh token without problems during that time./reloadin the stuck session did not help. The next prompt still failed withrevoked, even thoughauth.jsonalready held the valid token that other sessions were using.Why (from reading the 0.10.0 bundle)
AuthStorage.getApiKeyWithSourceTokenuses the in-memorythis.data[provider]. It reads disk (refreshOAuthTokenWithLock) only when the in-memoryexpirestime has passed. A token that another process has rotated or revoked still looks valid to this worker.authfailure,completeWithProviderRetry/_handleRetryableErrorretry once with the same credential.isPermanentProviderFailureKindreturns true onceretriesPerformed > 0 && kind === "auth". Then_markProviderAuthStaleForRetryFailuremarks the source stale. Nothing re-readsauth.jsonor forces a refresh./reload(AgentSession.reload→authStorage.reload()) should have loaded the fresh token, but the worker still sent a revoked one. So the token may also be cached somewhere else, or the stale marking may interfere. I have not confirmed which.Expected
When an OAuth credential gets a 401 (
revoked/expired):auth.jsonunder its lock. If the access token on disk is different, retry with it.expireshas not passed, save the result, then retry once./reloadshould be enough to recover in any case.Workaround
Restart the stuck session worker (kill its process; the daemon starts it again for the next due heartbeat).
/reloaddoes not work.Related: #1768 (refresh tokens rotate on use), #2297 / #2093 (tail 401 after long waits).
All reactions