Skip to content

smoke-codex: api-proxy records zero token-usage records, failing verify_token_usage sanity check #6537

Description

@lpcox

Summary

The smoke-codex workflow failed in the verify_token_usage job even though the Codex agent ran successfully. The api-proxy persisted zero per-response token-usage records for a Codex/OpenAI /responses streaming run, so scripts/ci/check-token-usage.js correctly failed with "No token-usage records found."

Error

Token-usage sanity check (engine: codex)
  token-usage.jsonl: /tmp/gh-aw-agent/sandbox/firewall/logs/api-proxy-logs/token-usage.jsonl
  agent_usage.json:  (not found)
  totals: responses=0 input=0 output=0 cache_read=0 cache_write=0
##[error]No token-usage records found. The agent produced no model requests, or the api-proxy failed to record usage.
Token-usage sanity check FAILED with 1 error(s).

What actually happened

The agent job itself passed. Codex ran fine and exited 0:

  • It routed through the OpenAI api-proxy: configured OPENAI_BASE_URL from /reflect for provider=openai: http://api-proxy:10000
  • Codex self-reported real usage: turn.completed usage input_tokens=256703, cached_input_tokens=231424, output_tokens=5297, reasoning_output_tokens=3521
  • It ran tools and posted a safe-output comment successfully.

But the api-proxy wrote no records to token-usage.jsonl. The in-job "Parse token usage for step summary" step confirms this independently: "No token usage data found, skipping summary." The downstream verify_token_usage check then saw an empty file and failed.

Root cause

The api-proxy's HTTP token tracker (containers/api-proxy/token-tracker-http.js) did not capture/persist any usage for this Codex/OpenAI /responses streaming response — an SSE usage-capture miss. This is the same fragile streaming-usage path recently touched by #6517 ("track token usage for aborted streaming responses") and #5262.

Not caused by PR #6528

The run surfaced on #6528, but that PR only changes containers/api-proxy/providers/copilot-byok.js (+ test) — the Copilot BYOK path. This smoke run used the OpenAI provider (API proxy enabled: OpenAI=true … Copilot=false), so Codex never executed the refactored code. smoke-codex passes on main and other branches around the same time, which points to an intermittent api-proxy usage-capture miss on the Codex SSE path rather than a PR regression.

Suggested next steps

  • Investigate OpenAI /responses SSE usage extraction in containers/api-proxy/token-tracker-http.js (extractUsageFromSseLine / stream-teardown finalization in extractUsageFromTrackedState) for cases where Codex tears down the socket after the final response.completed event without the tracker capturing the usage payload.
  • Consider adding a diagnostic assertion that logs HTTP_TRACK_END has_usage=false prominently, and/or a fallback that reconciles against the agent-reported usage.
  • Determine whether the miss is deterministic on this branch (re-run needed) or a flake; if it recurs only here while main stays green, dig deeper — but the PR diff cannot be the trigger.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions