Concurrent ucode opencode/gemini sessions race on the shared OAuth cache during --force-refresh, revoking the token family → Forbidden: Invalid Token

[jesco39]

Summary

Running multiple concurrent ucode opencode (or ucode gemini) sessions against the same workspace/profile intermittently produces Forbidden: Invalid Token on all sessions, immediately after a background token refresh that appeared to succeed. The freshly written token is valid in isolation (a direct curl to the AI Gateway with it returns 200), but requests still 403 shortly after.

Root cause is a cross-process race on the shared Databricks OAuth cache during --force-refresh, with no locking anywhere in ucode.

This is distinct from #98 (which was about x-api-key vs Authorization header routing). Here the header routing is correct; the token itself gets revoked by a concurrent refresh.

Environment

• ucode installed via uv tool
• OpenCode 1.17.10 (stock), macOS (arm64)
• ~/.databrickscfg uses auth_type = databricks-cli with auth_storage = secure (macOS Keychain, single databricks-cli item)
• Multiple ucode opencode sessions running simultaneously (normal daily workflow — 4–5 terminals)

Mechanism

src/ucode/agents/opencode.py launches a per-process daemon thread _refresh_forever, which every TOKEN_REFRESH_INTERVAL_SECONDS (1800s) calls _refresh_token_once(..., force_refresh=True) → get_databricks_token(..., force_refresh=True) → databricks auth token --force-refresh (src/ucode/databricks.py:811-812). agents/gemini.py has the identical pattern.

Every concurrent ucode process runs its own independent timer against the same OAuth cache (the Keychain item). grep -rn "fcntl|flock|Lock|filelock" src/ucode/ returns nothing — there is no cross-process (or in-process) serialization around the refresh.

Databricks OAuth rotates the refresh token on redemption. When two sessions' 30-minute timers land close together, both redeem the same refresh token concurrently; the second presents an already-rotated (consumed) refresh token, which trips the OAuth server's refresh-token-reuse detection and revokes the whole token family — invalidating the access token that was just issued and written into opencode.json. Hence "Forbidden" right after a "successful" refresh.

Reproduction

1. Configure ucode for opencode against a v2 AI Gateway workspace (Keychain-backed databricks-cli profile).
2. Launch 4–5 concurrent ucode opencode sessions and leave them running past the 30-minute refresh boundary so multiple refresh timers overlap.
3. Observe intermittent Forbidden: Invalid Token across sessions after a refresh tick. Recovery requires databricks auth logout && databricks auth login to reset the token family.

(The race is timing-dependent; overlapping timers across ≥2 long-lived sessions make it likely within a few refresh cycles.)

Impact

Multiple concurrent sessions is a common workflow. Today the only recovery is a manual re-login, and it recurs.

Proposed fix (two independent axes)

Axis 1 — serialize refreshes (fixes the revocation). Wrap the --force-refresh path in get_databricks_token (databricks.py) with a cross-process fcntl.flock on a per-workspace+profile lockfile (e.g. under the ucode app dir). The winner refreshes; others block briefly, then read the now-fresh cache and skip their own force-refresh if it was refreshed within the last few seconds. Benefits every agent (opencode + gemini share the pattern).

Axis 2 — stop baking a static, expiring token into a shared file for opencode. OpenCode has no apiKeyHelper and resolves {env:}/{file:} only at startup, so the 30-min file rewrite never reaches a running session anyway. A ucode-managed OpenCode plugin using the per-request chat.headers hook can set Authorization from a freshly sourced token on each request (verified empirically against stock 1.17.10: a plugin-set Authorization overrides the static provider.options.headers.Authorization on the wire). Combined with Axis 1's lock, the token source stays serialized. This removes the shared-file-rewrite mechanism for opencode entirely.

Happy to contribute a PR for both axes (with tests + ruff) if the maintainers agree with the direction. Wanted to confirm awareness and preferred approach first.

[jesco39]

Update: I thought this was just an issue with multiple sessions, but seems to also be an issue with even 1 long running session.