feat(responses): stream-teed credit reconciliation for native passthrough

[HaruHunab1320]

Summary

Follow-up to #427, addressing the "settlement fires before stream consumed" finding from that PR's review. Stacked on top of fix/responses-native-passthrough — merge #427 first, then this rebases onto dev cleanly.

#427 settled the credit reservation to the reserved estimate on every request regardless of actual usage. A Codex turn that emitted 200 output tokens was charged at the same rate as one that emitted 4000, as long as both fit under the reserved upper bound. This PR fixes that by extracting real usage from the SSE stream and reconciling on stream close.

What changed

New: packages/lib/utils/responses-stream-reconcile.ts

Pure helper that wraps a ReadableStream<Uint8Array> so bytes flow to the client unchanged while the wrapper scans for the terminal response.completed SSE event and extracts response.usage. Exactly one terminal callback per stream lifecycle (end | cancel | error), regardless of how the stream ends.

const trackedBody = wrapWithUsageExtraction(upstream.body, (usage, reason) => {
  if (usage) reconcile(calculateCost(model, provider, usage.inputTokens, usage.outputTokens));
  else       reconcile(reservedAmount);
});

Design constraints (all enforced by tests):

• Byte-exact passthrough — bytes flow in the exact order and size the upstream produced them. No batching, rewriting, or buffering beyond a single SSE frame for parser bookkeeping.
• Parse errors are swallowed — a malformed frame must never break the forward path.
• Chunk-boundary handling — SSE frames split mid-JSON across multiple reads are reassembled correctly.
• Exactly-one terminal callback — fires on end, cancel, or error. A buggy callback that throws does not break the stream.
• Pull-based — upstream is only drained when the client reads, matching direct-proxy semantics.
• [DONE] sentinel ignored.

app/api/v1/responses/route.ts — wired into the passthrough

Replaces the previous fire-and-forget background settle(reservedAmount) with:

const reconciledBody = wrapWithUsageExtraction(upstreamResponse.body, (usage, reason) => {
  logger.debug(...);
  void runReconciliation(usage);
});

return new Response(reconciledBody ?? upstreamResponse.body, { ... });

runReconciliation():

1. If usage is null (no response.completed seen) → settle to reserved (same as pre-stream-wrap behavior)
2. If usage is present → calculateCost(model, provider, inputTokens, outputTokens) then settle to min(computed, reserved)
3. If calculateCost throws → fall back to settling at reserved

Trade-offs (documented in inline comments)

• Cap at reserved, not over-collect. If the model somehow runs hotter than the reservation covered, we can't retroactively collect more from this path — the reserved amount is the upper bound. Anything beyond would need a separate post-hoc ledger entry which this PR doesn't implement.
• Cancel-before-completed charges the upper bound. Client aborts (Codex CLI Ctrl-C, tab close) before response.completed arrives settle to reserved. That's the same behavior as before this PR for the cancel case — the 50% safety buffer in estimateRequestCost is still our protection there.
• No impact when upstream.body === null. We fall back to an immediate synchronous settle-to-reserved so we don't strand the reservation.

Tests

14 unit tests — packages/tests/unit/utils/responses-stream-reconcile.test.ts

Passthrough fidelity

• Forwards bytes unchanged across multiple chunks
• Handles SSE frames that span chunk boundaries
• Handles the [DONE] sentinel without crashing
• Malformed JSON in a data: line does not break the forward path

Usage extraction

• Extracts input_tokens / output_tokens from response.completed
• Extracts cached_tokens and reasoning_tokens when present
• Omits cached/reasoning fields when zero or missing
• Returns null when no response.completed event arrives
• Defaults missing numeric fields to 0 (zero-usage event is still a reconciliation)

Termination

• Fires onComplete exactly once on normal close
• Fires with 'cancel' when reader is cancelled mid-stream
• Still extracts usage when cancel fires AFTER response.completed
• Fires with 'error' when the upstream throws
• A throwing onComplete callback does not break the stream

4 new integration tests — packages/tests/unit/api/v1-responses-route.test.ts

• Reconciles to actual cost when response.completed reports usage
• Caps actual cost at reserved when model over-runs the estimate
• Falls back to reserved when no response.completed arrives
• Existing mockReconcileCredits assertion tests updated to drain the wrapped body (pull-based wrapper gates the callback on reader progress)

Test totals: 48 passing across the two affected suites (14 unit + 34 route, was 31).

Test plan

• bun test packages/tests/unit/utils/responses-stream-reconcile.test.ts — 14/14
• bun test packages/tests/unit/api/v1-responses-route.test.ts — 34/34
• bunx tsc --noEmit clean on all changed files (only pre-existing bs58 errors)
• bun run lint clean (biome auto-fix applied)
• Merge fix(responses): native Responses-API passthrough for gpt-5.x + Codex CLI #427 first (this branch is stacked on it)
• Deploy to dev and test end-to-end with Codex CLI via milady's PARALLAX_LLM_PROVIDER=cloud flow, confirm small turns get reconciled down from the estimate and large turns stay capped at reserved

Follow-ups (not in this PR)

• Post-hoc over-run ledger — if the model uses more than reserved, we silently cap. A proper fix would emit a separate ledger entry for the delta, visible in the user's billing dashboard. Out of scope here.
• Per-model usage details — we extract cached_tokens and reasoning_tokens from the usage object but don't yet use them in calculateCost (which is input/output only). When cost calculation grows to support cached/reasoning tiers, the plumbing is already in place.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
eliza-cloud-v2	Ready	Preview, Comment	Apr 8, 2026 3:21am

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 6c0736a4-db9e-4651-b4cc-7edd87b0f304

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/responses-passthrough-stream-reconciliation

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[HaruHunab1320]

Closing — code already landed on dev via Shaw's manual merge 44a29116 ("merge: integrate PR #428 responses stream reconciliation").

GitHub kept this PR in OPEN state because the merge was done with git merge rather than the GitHub "Merge pull request" button, and because this branch's base was fix/responses-native-passthrough (the #427 branch, now deleted). The head commit 542e9607 is reachable from origin/dev — verified via git branch -r --contains — and the merge commit brought in all four files from this PR (app/api/v1/responses/route.ts, packages/lib/utils/responses-stream-reconcile.ts, both test files).

Shaw also added a follow-up hardening commit df4e31b6 ("fix: harden responses passthrough and managed env config") on top of both merges.

Closing to keep the open-PR list clean. No further action needed on this branch.