Found while running the dbt-optimizer live eval (PR #1092) — the eval spawns the compiled binary in an isolated HOME with env-only model credentials, which is exactly the headless/CI posture. Eight runs produced six distinct product defects before a single model call succeeded:
run exits 0 on fatal errors. Error: Token refresh failed: 400 and Model not found: ... both print to the transcript and exit 0. Headless consumers cannot detect failure by exit code.- Errored CLI lingers as a dangling process. After a provider-lookup error, the process survived
spawnSync timeout (SIGTERM and SIGKILL configured) — observed alive ~2h and ~69m in two runs until bun test force-killed it.
- Silent stream hang on google-vertex-anthropic. With valid ADC (verified via
gcloud auth application-default print-access-token), the session starts, the title-agent call is issued... and nothing ever arrives — no error, no timeout, no finish. Logs end at session.updated.
- Google provider env split-brain. Provider detection accepts
GEMINI_API_KEY, but the SDK call then demands GOOGLE_GENERATIVE_AI_API_KEY — detection succeeds, invocation fails.
- Vertex provider env split-brain. The models.dev env list advertises
GOOGLE_VERTEX_PROJECT/GOOGLE_VERTEX_LOCATION, but the loader (provider.ts:629) only detects via GOOGLE_CLOUD_PROJECT/GCP_PROJECT/GCLOUD_PROJECT; GOOGLE_VERTEX_PROJECT is an output var only. Env-only vertex configuration per the documented names cannot work.
- OAuth auth store is location-bound. Copying
auth.json to a new HOME yields Token refresh failed: 400 (refresh-token rotation), so OAuth accounts have no headless path at all; and per (1) that failure is invisible to exit-code checks.
Repro for each is in the PR #1092 eval harness: OPTIMIZER_LIVE_EVAL=1 + OPENCODE_TEST_CLI + the documented env vars; transcripts captured via OPTIMIZER_EVAL_TRANSCRIPT.
Suggested severity: (1) and (3) first — they make headless usage silently unreliable; (4)/(5) are one-line env additions; (2) needs a process-exit audit on the error path; (6) needs a documented headless auth story (API-key path or device-flow re-auth).
Found while running the dbt-optimizer live eval (PR #1092) — the eval spawns the compiled binary in an isolated HOME with env-only model credentials, which is exactly the headless/CI posture. Eight runs produced six distinct product defects before a single model call succeeded:
runexits 0 on fatal errors.Error: Token refresh failed: 400andModel not found: ...both print to the transcript and exit 0. Headless consumers cannot detect failure by exit code.spawnSynctimeout (SIGTERM and SIGKILL configured) — observed alive ~2h and ~69m in two runs untilbun testforce-killed it.gcloud auth application-default print-access-token), the session starts, the title-agent call is issued... and nothing ever arrives — no error, no timeout, no finish. Logs end atsession.updated.GEMINI_API_KEY, but the SDK call then demandsGOOGLE_GENERATIVE_AI_API_KEY— detection succeeds, invocation fails.GOOGLE_VERTEX_PROJECT/GOOGLE_VERTEX_LOCATION, but the loader (provider.ts:629) only detects viaGOOGLE_CLOUD_PROJECT/GCP_PROJECT/GCLOUD_PROJECT;GOOGLE_VERTEX_PROJECTis an output var only. Env-only vertex configuration per the documented names cannot work.auth.jsonto a new HOME yieldsToken refresh failed: 400(refresh-token rotation), so OAuth accounts have no headless path at all; and per (1) that failure is invisible to exit-code checks.Repro for each is in the PR #1092 eval harness:
OPTIMIZER_LIVE_EVAL=1+OPENCODE_TEST_CLI+ the documented env vars; transcripts captured viaOPTIMIZER_EVAL_TRANSCRIPT.Suggested severity: (1) and (3) first — they make headless usage silently unreliable; (4)/(5) are one-line env additions; (2) needs a process-exit audit on the error path; (6) needs a documented headless auth story (API-key path or device-flow re-auth).