feat(providers): fill the Usage seam per provider (#624 Phase 2) by sroussey · Pull Request #685 · workglow-dev/libs

sroussey · 2026-08-05T23:25:38Z

Closes #624

Stacked PR — merge after #641

This branches from ai-provider-cache-checkpoints (PR #641), not main, and targets that branch. It must merge after #641.

Phase 1 (#680, already on main) added the generic seam — Usage, USAGE_OUTPUT_KEY, mergeUsage, and usage as an optional sibling of data on StreamFinish — with every provider emitting usage: undefined. This PR is the per-provider fill-in: each run-fn maps its own terminal frame into Usage and attaches it to the finish event it already emits. The consumer side (StreamEventAccumulator.applyUsage / StreamProcessor) is untouched.

usage is always attached as a sibling of data, never inside it — folding token counts into data would break the {}-for-delta-streams, full-Output-for-one-shot, and data.object-for-json-mode conventions all at once.

The `undefined` vs `0` rule

undefined means "the provider did not report this figure". It never means zero.

A model that billed 0 cached tokens and a model that says nothing about caching are different facts, and collapsing them to 0 silently understates spend — the reader of a cost report cannot tell "this request used no cache" from "we have no idea whether it did". So an absent, null, NaN, or non-numeric wire field maps to undefined, while a genuinely reported 0 survives as 0. Providers with no usage at all leave it absent entirely.

Usage is also always read from the provider's own terminal frame — never derived by counting our own emitted output, which would drift from what the provider actually bills.

Per-provider mapping

Provider	Source frame	input	output	cached	cacheWrite	reasoning	total	extra
anthropic	`message_start` + `message_delta`	`input_tokens`	`output_tokens`	`cache_read_input_tokens`	`cache_creation_input_tokens`	`output_tokens_details.thinking_tokens`	—	—
openai (Responses)	`response.completed`	`input_tokens`	`output_tokens`	`input_tokens_details.cached_tokens`	`input_tokens_details.cache_write_tokens`	`output_tokens_details.reasoning_tokens`	`total_tokens`	—
openai (embeddings)	non-streaming result	`usage.prompt_tokens`	—	—	—	—	`usage.total_tokens`	—
google-gemini	last non-null `chunk.usageMetadata`	`promptTokenCount`	`candidatesTokenCount`	`cachedContentTokenCount`	—	`thoughtsTokenCount`	`totalTokenCount`	—
ollama	`done: true` chunk	`prompt_eval_count`	`eval_count`	—	—	—	—	—
xai	`include_usage` frame	`prompt_tokens`	`completion_tokens`	`prompt_tokens_details.cached_tokens`	—	`completion_tokens_details.reasoning_tokens`	`total_tokens`	—
deepseek	`include_usage` frame	`prompt_tokens`	`completion_tokens`	`prompt_cache_hit_tokens`	—	`completion_tokens_details.reasoning_tokens`	`total_tokens`	`promptCacheMissTokens`
openrouter	`include_usage` frame	`prompt_tokens`	`completion_tokens`	`prompt_tokens_details.cached_tokens`	—	`completion_tokens_details.reasoning_tokens`	`total_tokens`	`cost`
llamacpp-server	`include_usage` frame	`prompt_tokens`	`completion_tokens`	`prompt_tokens_details.cached_tokens`	—	`completion_tokens_details.reasoning_tokens`	`total_tokens`	—
huggingface-inference	terminal chunk, if volunteered	`prompt_tokens`	`completion_tokens`	`prompt_tokens_details.cached_tokens`	—	`completion_tokens_details.reasoning_tokens`	`total_tokens`	—

Providers that report nothing — `usage` stays absent

Provider	Why
node-llama-cpp	Runs the model locally; its prompt API surfaces no token accounting.
huggingface-transformers	transformers.js runs in-process and reports no counts.
chrome-ai	The Chrome built-in AI API exposes only quota measures (`inputUsage`), not billing counters.
tf-mediapipe	Vision pipelines; no token concept.

These are left absent with a short comment recording why, so the next reader does not "fix" it by emitting zeros or by counting our own emitted tokens.

Notes on specific decisions

Anthropic reports no total. Synthesizing one from the parts would misreport cache-discounted input as if it had been billed in full, and Usage.total is documented as never synthesized. Both frames restate cumulative figures, so the collector keeps the newest value per field rather than summing; a null on message_delta means "not restated here" and leaves the earlier value intact rather than erasing it.
DeepSeek's prompt_cache_miss_tokens has no normalized slot but is what makes its cache-miss-priced cost reconstructable, so it rides in extra.
OpenRouter's cost is a price, not a token count. It is worth carrying because OpenRouter routes to whichever upstream is cheapest, which the caller cannot reconstruct locally.
Ollama field names were verified against the installed SDK's ChatResponse / GenerateResponse types rather than taken on trust.

Shared plumbing

packages/ai/src/provider-utils/UsageMapping.ts normalizes the two OpenAI wire shapes and exposes the primitives provider packages compose their quirks from (toUsageCount, usageOrUndefined, usageExtra, OPENAI_STREAM_USAGE_OPTIONS).

Both shared accumulators now return the usage they read instead of discarding it:

accumulateOpenAIResponsesStream handles response.completed (plus the incomplete / failed terminals, which still consumed tokens) rather than dropping them into the default: arm. One change covers OpenAI TextGeneration, ToolCalling, Rewriter and Summary.
accumulateOpenAIChatStream reads chunk.usage before skipping a choice-less chunk, and takes an optional mapper so DeepSeek/OpenRouter can supply their own.

`include_usage` safety

xAI, DeepSeek, OpenRouter and llama.cpp-server needed stream_options: { include_usage: true } added to the request. That opt-in appends a final chunk whose choices array is empty — exactly the shape the *_StreamChunkSafety suites exist to guard. Every delta read already optional-chains chunk.choices?.[0], and usage is read before the choice-less chunk is skipped. Those suites were extended rather than duplicated.

llama.cpp-server parses SSE itself, so IChatCompletionDelta gained a usage field and the parser now admits a usage-only frame on its own merit — it previously required a content delta, tool calls, or a finish reason, which silently dropped the usage chunk.

Tests

A shared contract assertion at packages/test/src/contract/ai-provider/assertions/usageNormalization.ts (sibling of sessionReuse.ts) holds every provider's mapper to the same rules: no payload maps to undefined, unreported counters come back undefined rather than 0, a genuinely reported 0 survives, and only numbers and undefined reach the normalized shape. ProviderUsageNormalization.test.ts runs all seven mappers through it, so the zero-vs-absent rule is enforced in one place instead of being restated per provider.

Per-provider suites were extended in place: Xai_StreamChunkSafety, OpenRouter_StreamChunkSafety, LlamaCppServer_TextGenerationStream, Ollama_TextGenerationStream, DeepSeek_ToolCalling, Anthropic_SamplingParams, and OpenAIShapedResponses.

Verification

All commands run against the committed branch state.

$ bun run build
 Tasks:    84 successful, 84 total
Cached:    84 cached, 84 total

$ bun run build:types
 Tasks:    41 successful, 41 total
Cached:    41 cached, 41 total

$ bun scripts/test.ts ai provider provider-nodellama vitest unit
 Test Files  1 failed | 63 passed (64)
      Tests  1 failed | 669 passed | 1 skipped (671)

$ bun scripts/test.ts provider-api vitest unit
 Test Files  38 passed (38)
      Tests  431 passed (431)

provider-api went from 413 to 431 tests; the ai/provider sweep gained the shared-contract and Responses-usage suites.

Inherited failure — not from this PR

The single failure is packages/test/src/test/ai-provider/OpenAIGeminiCheckpointParams.test.ts → "evicts and retries inline once on a NOT_FOUND".

It already fails on the base branch: commit 5d9088f3 fix(gemini): tighten reactive CachedContent NOT_FOUND matcher (the rebased 3206e7de) tightened the CachedContent NOT_FOUND matcher without updating that older test. 5d9088f3 is the second commit of ai-provider-cache-checkpoints, and this PR's diff touches neither Gemini_CachedContentFallback.ts nor that test file — the stack trace runs entirely through generateGeminiStreamWithCacheFallback, which is untouched here. It is reported on #641 and awaiting the owner's decision; it is deliberately not fixed in this PR.

Only the unit test kind was run. Integration suites were not run locally, since the preloader unlocks real API keys and test-vitest-ai-provider-api makes real billable gpt-image-2 calls.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PwyJuFrJnibKvrrk8Fa4Fn

Generated by Claude Code

sroussey · 2026-08-05T23:30:17Z

CI test-vitest-unit is red on cd5f3a0f, with exactly one failure, inherited from the base branch:

Test Files  1 failed | 389 passed | 1 skipped (391)
     Tests  1 failed | 4872 passed | 43 skipped (4916)

The failure is packages/test/src/test/ai-provider/OpenAIGeminiCheckpointParams.test.ts → "evicts and retries inline once on a NOT_FOUND", the same one reported on #641. Its stack runs entirely through generateGeminiStreamWithCacheFallback, and this PR's diff touches neither that function nor that test.

Re-running will not help — it is deterministic, not a flake. It clears only when the #641 question is decided:

commit 5d9088f3 ("tighten reactive CachedContent NOT_FOUND matcher") made the matcher require a cache-scoped signal — a cachedContents/… ResourceInfo or cachedContent wording — specifically so a model 404 is not mistaken for a cache 404;
the older test still throws { status: 404, code: "NOT_FOUND", message: "cache not found" }, which carries neither signal, so the tightened matcher deliberately declines to evict and retry.

Either the fixture should become cache-scoped, or the matcher is stricter than the real Gemini error shape warrants. That second possibility is the reason this has not been "fixed" unilaterally: if a genuine cache eviction really does surface as an unscoped NOT_FOUND, loosening the fixture would paper over a production bug in which a stale cachedContent never triggers the inline-replay fallback.

Blocked on that decision. Everything else in this PR is green — build 84/84, build:types 41/41, and provider-api up from 413 to 431 tests.

Generated by Claude Code

github-actions · 2026-08-05T23:58:05Z

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	61.59%	29200 / 47406
🔵	Statements	61.47%	30250 / 49207
🔵	Functions	62.59%	5558 / 8880
🔵	Branches	50.33%	14701 / 29205

File Coverage

File	Stmts	Branches	Functions	Lines	Uncovered Lines
Changed Files
providers/huggingface-transformers/src/ai/common/HFT_TextGeneration.ts	1.42%	0%	0%	1.49%	32-193
providers/node-llama-cpp/src/ai/common/LlamaCpp_Chat.ts	80.59%	66.66%	100%	82.25%	111-119, 127, 134, 143, 176-180

Generated in workflow #2872 for commit acfcfd4 by the Vitest Coverage Report Action

Adds the mapping layer every provider fills the Phase 1 `Usage` seam through, plus the contract that holds them to one rule. `UsageMapping.ts` normalizes the two OpenAI wire shapes — chat completions (`prompt_tokens`/`completion_tokens`) and Responses (`input_tokens`/`output_tokens`, cache reads and writes nested under `input_tokens_details`) — and exposes the primitives provider packages compose their own quirks from: `toUsageCount`, `usageOrUndefined`, `usageExtra`, and the `OPENAI_STREAM_USAGE_OPTIONS` request fragment. The governing rule is that `undefined` means "the provider did not report this", never "zero": an absent, null, NaN or non-numeric wire field stays `undefined`, while a genuinely reported `0` survives as `0`. Collapsing the two would silently understate spend, since a model that billed no cached tokens and a model that says nothing about caching are different facts. Both shared stream accumulators now return the usage they read off the provider's own terminal frame instead of discarding it: - `accumulateOpenAIResponsesStream` handles `response.completed` (and the `incomplete`/`failed` terminals, which still consumed tokens) rather than dropping them into the `default:` arm. - `accumulateOpenAIChatStream` reads `chunk.usage` BEFORE skipping a choice-less chunk — `include_usage` delivers usage on a final frame whose `choices` array is empty. It takes an optional mapper so a provider reporting extra counters can supply its own. Callers attach the result to their `finish` event as a sibling of `data`, never inside it. The shared contract assertion feeds each provider's mapper the same frames: no payload maps to `undefined`, unreported counters come back `undefined` rather than `0`, a real `0` survives, and only numbers and `undefined` reach the normalized shape. Claude-Session: https://claude.ai/code/session_01PwyJuFrJnibKvrrk8Fa4Fn

Anthropic splits its accounting across two frames — `message_start` carries the prompt side (`input_tokens`, `cache_read_input_tokens`, `cache_creation_input_tokens`) while `message_delta` carries the running completion side — and both restate CUMULATIVE figures. `createAnthropicUsageCollector` keeps the newest number stated per field rather than summing restatements, so it never derives a count of its own. A `null` on `message_delta` means "not restated here" (older API versions null the prompt-side fields there), so it leaves the earlier value intact instead of erasing it; a field no frame ever reported stays `undefined`. Mapping: `input_tokens` → input, `output_tokens` → output, `cache_read_input_tokens` → cached, `cache_creation_input_tokens` → cacheWrite, `output_tokens_details.thinking_tokens` → reasoning. `total` is deliberately left unreported — Anthropic states none, and synthesizing one from the parts would misreport cache-discounted input as if it had been billed in full. Wired into TextGeneration, ToolCalling, StructuredGeneration, TextSummary and TextRewriter, all of which already iterate the raw SDK event stream. Claude-Session: https://claude.ai/code/session_01PwyJuFrJnibKvrrk8Fa4Fn

…ings The Responses API reports usage unprompted on its terminal `response.completed` event, which the shared accumulator was discarding. Reading it there covers TextGeneration, ToolCalling, Rewriter and Summary at once; StructuredGeneration runs its own loop and reads the same terminal events directly. Mapping: `input_tokens` → input, `input_tokens_details.cached_tokens` → cached, `input_tokens_details.cache_write_tokens` → cacheWrite, `output_tokens` → output, `output_tokens_details.reasoning_tokens` → reasoning, `total_tokens` → total. `OpenAI_TextEmbedding` is non-streaming: it maps `usage.prompt_tokens` → input and `usage.total_tokens` → total. The embeddings endpoint reports no completion, cache or reasoning counters, so those stay unreported rather than being zeroed. Claude-Session: https://claude.ai/code/session_01PwyJuFrJnibKvrrk8Fa4Fn

Gemini attaches `usageMetadata` to streamed chunks, restating cumulative counts as generation proceeds. Each run-fn keeps the last non-null block and maps it once at finish, so nothing is summed locally. Mapping: `promptTokenCount` → input, `candidatesTokenCount` → output, `cachedContentTokenCount` → cached, `thoughtsTokenCount` → reasoning, `totalTokenCount` → total. `promptTokenCount` already includes the cached-content tokens that `cachedContentTokenCount` breaks out, matching the input/cached relationship the other providers use. Wired into TextGeneration, ToolCalling, StructuredGeneration, TextSummary and TextRewriter. Claude-Session: https://claude.ai/code/session_01PwyJuFrJnibKvrrk8Fa4Fn

Ollama reports counts only on the final `done: true` chunk of a chat stream: `prompt_eval_count` is the prompt side and `eval_count` the generated side (field names verified against the installed SDK's `ChatResponse`/`GenerateResponse` types, not assumed). Intermediate chunks omit both, so `mapOllamaUsage` returns `undefined` for anything but the terminal chunk rather than mistaking an absent count for zero. Ollama runs locally and reports no cache, reasoning or total counters, so those slots stay unreported. Wired into TextGeneration, ToolCalling, StructuredGeneration, TextSummary and TextRewriter. Claude-Session: https://claude.ai/code/session_01PwyJuFrJnibKvrrk8Fa4Fn

…iders xAI, DeepSeek, OpenRouter and llama.cpp-server send no usage at all unless the request asks for it, so each streaming chat request now carries `stream_options: { include_usage: true }`. That opt-in appends a final chunk whose `choices` array is EMPTY, which is exactly the shape the `*_StreamChunkSafety` suites exist to guard. Every delta read already optional-chains `chunk.choices?.[0]`, and usage is read before the choice-less chunk is skipped. Standard mapping for all four: `prompt_tokens` → input, `completion_tokens` → output, `prompt_tokens_details.cached_tokens` → cached, `completion_tokens_details.reasoning_tokens` → reasoning, `total_tokens` → total. Two providers report more than that common set, kept in their own packages rather than leaking into the shared helper: - DeepSeek splits the prompt into `prompt_cache_hit_tokens` (the cache-read figure, so it fills `cached`) and `prompt_cache_miss_tokens` (no normalized slot, carried in `extra` — it is what makes DeepSeek's cache-miss-priced cost reconstructable). - OpenRouter reports `cost`, the credits actually charged. That is a price rather than a token count, so it rides in `extra`; it is worth carrying because OpenRouter routes to whichever upstream is cheapest, which the caller cannot reconstruct locally. llama.cpp-server parses SSE itself, so `IChatCompletionDelta` gains a `usage` field and the parser admits a usage-only frame on its own merit — it previously required a content delta, tool calls or a finish reason, which silently dropped the usage chunk. Hugging Face Inference gets no `include_usage` opt-in (it routes to third-party providers whose support varies) but now forwards usage when the upstream volunteers it, instead of discarding the accumulator's return value. Claude-Session: https://claude.ai/code/session_01PwyJuFrJnibKvrrk8Fa4Fn

node-llama-cpp, HuggingFace Transformers and Chrome built-in AI surface no token accounting: the first two run the model in-process, and the Chrome API exposes only quota measures (`inputUsage`), not billing counters. They therefore leave `usage` absent. Comments record why, so the next reader does not "fix" it by emitting zeros or by counting our own emitted tokens — a local estimate dressed up as a provider fact. Claude-Session: https://claude.ai/code/session_01PwyJuFrJnibKvrrk8Fa4Fn

Extends the existing per-provider stream-shape suites rather than adding parallel ones, since the terminal `choices: []` usage frame is precisely the chunk shape those suites already guard against. - Xai / OpenRouter `StreamChunkSafety`: assert the request carries `stream_options`, that the terminal frame maps onto `finish.usage`, that OpenRouter's `cost` lands in `extra`, and that a usage frame trailing a structured stream still assembles its object. - `LlamaCppServer_TextGenerationStream`: drives the bespoke SSE parser with a usage-only frame — the regression that motivated admitting a choice-less chunk — and checks it is not mistaken for content. - `Ollama_TextGenerationStream`: the `done: true` chunk's counts, an absent-usage stream, and that counts on non-terminal chunks are ignored. - `DeepSeek_ToolCalling`: the cache hit/miss split end to end. - `Anthropic_SamplingParams`: the two-frame collector — cumulative restatement, `null` treated as "not restated" rather than an erasure, thinking tokens as reasoning. - `OpenAIShapedResponses`: usage off `response.completed`, plus the `incomplete`/`failed` terminals that still consumed tokens. `ProviderUsageNormalization` runs every provider's mapper through the shared contract, so the zero-vs-absent rule is enforced in one place instead of being restated per provider. Claude-Session: https://claude.ai/code/session_01PwyJuFrJnibKvrrk8Fa4Fn

…ime bundles The provider ships ./ai and ./ai-runtime as separate dist entry points (dist/ai.js, dist/ai-runtime.js). Both independently import LlamaCpp_Runtime.ts, so its module-level `llamaCppSessions = new Map()` was evaluated twice in-process, producing two distinct Map instances under inline (non-worker) usage. LlamaCppQueuedProvider.disposeSession's local-delete fallback (bundled into ./ai) mutated its own copy while registerLlamaCppInline/setLlamaCppSession (bundled into ./ai-runtime) wrote to the other, so the fallback silently no-opped instead of disposing the session and freeing its LlamaContext sequence slot. Keyed the map behind a Symbol.for(...) on globalThis, the same pattern already used by PortCodecRegistry and the DI Container for state shared across split entry points. Caught by LlamaCpp_SessionDispose.test.ts (added in #706); CI never ran on that PR due to a GitHub Actions outage, so this surfaced only when run locally as part of merging #685 on top of #706.

* feat(ai): add cache.checkpoint capability and CheckpointRegistry * refactor(ai): widen run-fn sessionId param to structured AiSessionContext Claude-Session: https://claude.ai/code/session_01DsBvrmfhe1aY63cjztn1WW * feat(ai): add CacheCheckpointTask with eager warm-up and chaining Claude-Session: https://claude.ai/code/session_01DsBvrmfhe1aY63cjztn1WW * feat(ai): checkpoint rewind/emit ports on ToolCallingTask Claude-Session: https://claude.ai/code/session_01DsBvrmfhe1aY63cjztn1WW * feat(ai): checkpoint ports on TextGenerationTask and AiChatTask Claude-Session: https://claude.ai/code/session_01DsBvrmfhe1aY63cjztn1WW * feat(anthropic): cache.checkpoint warm-up and checkpoint-boundary cache_control Claude-Session: https://claude.ai/code/session_01DsBvrmfhe1aY63cjztn1WW * feat(hft): cache.checkpoint warm-up, emit snapshots, prefix re-encode fallback Claude-Session: https://claude.ai/code/session_01DsBvrmfhe1aY63cjztn1WW * fix(hft): render checkpoint prefix tools via mapHFTTools so warm-up matches consuming tokenization * feat(llamacpp): cache.checkpoint warm-up via preloadPrompt and checkpoint re-keying * fix(llamacpp): release acquired sequences when checkpoint preload or emit-only generation throws Claude-Session: https://claude.ai/code/session_01DsBvrmfhe1aY63cjztn1WW * test(ai): checkpoint chaining and branching integration tests; document cache checkpoints * fix(ai): final review fixes — HFT text-gen checkpoint handling, lifecycle docs, emit-session cleanup Claude-Session: https://claude.ai/code/session_01DsBvrmfhe1aY63cjztn1WW * fix(ai): restore run-scoped checkpoint disposal via ResourceScope Claude-Session: https://claude.ai/code/session_01DsBvrmfhe1aY63cjztn1WW * chore: re-baseline typecheck budgets for cache-checkpoint type growth packages/workglow grew 157 -> 242 instantiations (+85 absolute) from the new checkpoint type exports flowing through the meta-package re-exports; small expected growth in packages/ai, packages/test, and the three touched providers. Also records providers/duckdb, which was absent from the baseline. Claude-Session: https://claude.ai/code/session_01DsBvrmfhe1aY63cjztn1WW * fix(ai): harden cache-checkpoint consumption after whole-branch review Correctness fixes from the branch code review: - HFT_ToolCalling / HFT_Chat: render the checkpoint prefix into the fed prompt (continuation) and attach prefix KV only under a prompt.startsWith(prefixPrompt) parity guard, falling back to a full re-encode — previously the prefix KV was attached to a prompt that never contained the warm-up rendering, positionally corrupting generation. - renderHftPrefixPrompt: no empty user turn for message-less prefixes, so the parity guard can actually hold for the common system+tools warm-up. - HFT fingerprint tool-session: warm only the shared tools+systemPrompt region instead of the full prompt, so a second tool task sharing the fingerprint no longer attaches a cache poisoned by the first task's turn. - LlamaCpp_TextGeneration: consuming a checkpoint takes sole ownership of the live session (map entry removed, disposed at turn end unless re-keyed for emit) — a live sequence mutates in place, so a second consumer or a kept parent previously saw the first consumer's tokens and two ids could alias (and double-dispose) one native sequence. - resolveCheckpointSession: gate checkpoint/emitCheckpoint on the provider actually serving cache.checkpoint — OpenAI-shaped providers previously ignored the prefix silently (context dropped, handles backed by nothing). - ToolCallingTask emit accumulator: upsert toolCalls by id (mirroring StreamProcessor) instead of replacing — OpenAI-shaped providers emit one single-element array per tool call, so parallel calls lost all but one. - Anthropic checkpoint params/replay: guard on the converted message array; an all-system prefix.messages converts to [] and crashed annotateLastBlock. - HFT emit-only checkpoints now attach an empty cache so the turn's KV is snapshotted under the emitted id instead of discarded. Cleanups: shared validateParentCheckpoint/mergeCheckpointPrefix helpers (CacheCheckpointTask no longer duplicates them), redundant _parent field removed, stale _computedSessionId cleared between reused-instance runs, upfront getJobInput gated to checkpoint runs in TextGenerationTask, disposeSession idempotency contract documented, and the new checkpoint interfaces use explicit '| undefined' optionals. Claude-Session: https://claude.ai/code/session_01DsBvrmfhe1aY63cjztn1WW * feat(ai): OpenAI and Gemini cache-checkpoint support OpenAI maps checkpoints onto its automatic prompt cache: the warm-up run-fn sends the prefix once (Responses API, minimal output) and consumers replay the prefix content ahead of their tail via mergeOpenAICheckpointPrefix — the derived prompt_cache_key (model + instructions + tools) aligns warm-up and consumers without coordination. Gemini maps checkpoints to explicit server-side CachedContent with a 1h TTL: the warm-up creates the cache (degrading gracefully below the minimum cacheable size), consumers reference it with tail-only requests when the call adds no system prompt or non-default tool choice (the API rejects systemInstruction/tools alongside cachedContent) and replay the prefix inline otherwise. The id → cache-name store lives in an SDK-free module so both provider shells can delete the cache eagerly on disposeSession (TTL is the backstop). Also advertises cache.checkpoint in capability inference for supporting Anthropic / OpenAI / Gemini model families — previously only inline model configs with hand-declared capabilities passed the task-level gate. Claude-Session: https://claude.ai/code/session_01DsBvrmfhe1aY63cjztn1WW * fix(ai): keep local per-turn KV snapshotting for checkpoint-seeded chats A checkpoint fed to AiChatTask must never make the chat slower than no checkpoint. Previously run-fns inferred immutable-checkpoint semantics from the bare presence of session.prefix, which disabled HFT's per-turn KV snapshotting for the chat's own session — every turn re-encoded the growing conversation. AiSessionContext gains ownedSession: the sessionId is the caller's own mutable session merely seeded from the prefix. AiChatTask sets it; HFT_Chat keys immutability (snapshot target, supersede delete) on prefix && !ownedSession so progressive snapshotting stays alive; LlamaCpp_Chat labels the seeded session progressive and LlamaCpp_TextGeneration never applies take-ownership stealing to an owned session. Claude-Session: https://claude.ai/code/session_01DsBvrmfhe1aY63cjztn1WW * fix(ai): preserve local chat system prompts with checkpoints * test(ai): cover local checkpoint prompt construction * fix(llamacpp): support checkpoints in tool calling * fix(llamacpp): preserve tool checkpoint history * fix(gemini): validate cached checkpoint tools * fix(gemini): normalize cached schema arrays * fix(gemini): preserve literal schema arrays * fix(gemini): complete schema canonicalization * fix(gemini): dispose checkpoint caches in workers * test(gemini): cover worker checkpoint disposal boundary * test(llamacpp): type tool checkpoint model fixture * test(gemini): type worker cache model fixture * test(gemini): make checkpoint tests runtime portable * test(gemini): inject fake client via runtime seam for checkpoint tests Replace the unreliable vi.mock("@google/genai") in the Gemini checkpoint params test with a runtime test seam (_testOnly.setGeminiClientForTests). Module-level SDK mocks cannot intercept the provider here: the workspace ships several @google/genai copies and the provider resolves it from a bundled dist file the mock never reaches. The seam works identically for src and dist and captures the requests the run-fns build without a live call. Surface it through both the /ai and /ai-runtime _testOnly barrels since dist splits them into separate bundles with independent module state. Gate the worker-boundary session-dispose case with it.skipIf(!isBun): it spawns a real .ts worker that needs Bun's native TS-worker execution and bun:test's mock.module. It runs under `bun test`; the sibling in-process case covers the dispose path under vitest. * fix(node-llama-cpp): render checkpoint prefix through the chat template The prior renderer flattened a checkpoint prefix to raw "role: text" strings via renderLlamaCppPrefixText and preloaded that text — bypassing the model's chat wrapper and silently dropping tool_use / tool_result blocks. Warm-up tokens no longer matched what the consumer's generateResponse produced, so cache reuse was defeated and any prior tool exchange in the prefix was lost by the missing-state fallback. Replace with a ChatHistoryItem[] renderer that reuses the existing convertMessagesToChatHistory pipeline (extracted as a pure helper that skips the trailing empty-user placeholder). Warm-up and every missing-state fallback (Chat, TextGeneration, ToolCalling) now call session.setChatHistory(history) then session.preloadPrompt("", { functions }), so the KV state is populated through the same wrapper the consumer uses. Tools are handed to preloadPrompt via buildChatModelFunctions so the wrapper embeds their descriptions identically. Deleted renderLlamaCppPrefixText so no future path can silently drop tool blocks. Test plan updated: flip the reconstructed-checkpoint preloadPrompts expectation to [""] and assert the setChatHistory payload; add a tool_use/tool_result preservation regression, plus dedicated unit tests for the new render helpers (system-only, tools-only, mixed text+tool_use, tool_result matching). Claude-Session: https://claude.ai/code/session_01413eP4Awsvn2HSXqr4MFYh * fix(ai): fail-closed model-key guard on cache checkpoints checkpointModelKey returns "" for a missing model_id, and validateParentCheckpoint short-circuited its mismatch check on either side being empty — so two keyless models on the same provider silently shared a fungible checkpoint slot (cross-model contamination) and an emit path could mint a slot with no identity at all. Add requireCheckpointModelKey (throws TaskConfigurationError for a keyless model) and route every mint / validate site through it: the parent-validate site (mismatch comparison is now unconditional as defense in depth), the resolveCheckpointSession pre-dispatch gate (so emit paths fail before createSession mints), and CacheCheckpointTask.prepareCheckpoint's mint site (the returned key is threaded straight into registerCheckpoint). Tests: dedicated requireCheckpointModelKey unit coverage plus a model-key fail-closed L2 describe covering all five paths (warm-up rejection with no run-fn invocation, parent-consume rejection before dispatch, keyless-vs-keyless slot contamination, emit path rejection before createSession, and a regression on the existing keyed-mismatch path). Claude-Session: https://claude.ai/code/session_01413eP4Awsvn2HSXqr4MFYh * fix(node-llama-cpp): serialize concurrent checkpoint consumers with an atomic steal get + delete on the session map bracketed async work in both _TextGeneration and _ToolCalling stream paths; two concurrent consumers of the same immutable checkpoint id both observed the same cached state and both called .generate() on the shared LlamaContextSequence — a live sequence advances in place, so the second consumer's turn was corrupted. withModelInUse is a refcount, not a mutex, so it did not close the window. Add stealLlamaCppSession(id) in LlamaCpp_Runtime — a synchronous Map.get + Map.delete that is race-free on JS's single thread — and route both stream paths through it when consuming an immutable checkpoint (isCheckpoint && !ownedSession). Under contention exactly one caller wins; losers observe undefined and re-encode via the existing missing-state fallback (which acquires its own sequence). The ownership-tracking flag is now decided at the same point as the steal-vs-get split, dropping the downstream flip-flop that redundantly deleted the map entry on a hit. AiChatTask's ownedSession mode is the caller's mutable session (never a checkpoint), so LlamaCpp_Chat continues to use the non-consuming getLlamaCppSession — documented on the lookup site so a future refactor does not "unify" the two paths back into a shared race. Tests: stealLlamaCppSession unit coverage (atomic get+delete on hit, no-op on unknown id); a race test in the ToolCalling checkpoint suite covering the two-consumer scenario (winner reuses warmed sequence, loser re-encodes with its own) plus an ownedSession regression asserting racing ownedSession runs do not evict each other's map entries; and a new TextGenerationCheckpoint fixture mirroring the ToolCalling one so the text-generation path has parity coverage. Claude-Session: https://claude.ai/code/session_01413eP4Awsvn2HSXqr4MFYh * fix(google-gemini): narrow cache-checkpoint warm-up catch The Gemini cache-checkpoint warm-up wrapped ai.caches.create in a blanket try/catch that logged and swallowed every error, so a caller-initiated abort, a 401/403 auth failure, a 429 quota, and a 5xx transport error all silently degraded to the same "no cache, replay inline" fate as a genuine prefix-too- small 400. It also failed to clean up when a resource had been minted but a downstream step (bookkeeping / write) threw, leaving a server-side CachedContent billing until TTL. Classify the error into abort / degrade / throw. Abort and throw rethrow (with a best-effort delete of any resource whose name we saw); only a 400 / INVALID_ARGUMENT whose message names the too-small-prefix / cache-unsupported condition degrades to inline replay. Track `createdName` across the try / catch so the partial-delete cleanup can fire. Gemini_CacheStore gets a matching `createdAtMs` field on each entry, an `isGeminiCacheEntryStale` helper (default 3_500_000 ms — a hair under the 1h TTL used at write time), and a `deleteGeminiCachedContentLocal` for runtime-only eviction. These land here so PR #641's follow-up (H2, the consume-side fallback) can chain on top. Claude-Session: https://claude.ai/code/session_01UC9LGu3iokhuhAB46yt1ee * fix(google-gemini): fall back to inline replay on stale / NOT_FOUND cachedContent Gemini's explicit CachedContent is TTL-bound (~1h at creation) and can also vanish server-side outside the consumer's control. The text.generation and tool-calling run-fns treated the cache handle as guaranteed to resolve: an entry near-TTL still went out with `cachedContent`, and a NOT_FOUND surfaced as a hard failure to the caller. Add Gemini_CachedContentFallback: a proactive stale check (evicts the runtime-local entry before the request, leaves the server-side to its own TTL) plus a reactive NOT_FOUND catch that evicts (locally + best-effort server delete), rebuilds the request inline (prefix + tail), and retries once. Refactor both run-fns to share `buildCachedRequest` / `buildInlineReplayRequest` locals so the retry rebuilds an identical shape. Streaming contract unchanged (providers still yield text-delta/object-delta then finish; no accumulation). Claude-Session: https://claude.ai/code/session_01UC9LGu3iokhuhAB46yt1ee * test(node-llama-cpp): stub setChatHistory on the LocalChatCheckpoint fake session The L1 checkpoint-prefix-render fix routes the missing-state fallback in LlamaCpp_Chat through session.setChatHistory(history) + preloadPrompt(""), matching the real LlamaChatSession API. The checkpoint mocks under ai-provider-nodellama were updated to stub setChatHistory, but the FakeLlamaChatSession in this ai-provider suite was missed, so the checkpoint-seeded owned-session case threw "setChatHistory is not a function". Add the no-op stub to align the fake with the real session. Claude-Session: https://claude.ai/code/session_01YMyipztz9RFDp4aZPSNqEV * fix(hft): add required cacheKey to session snapshots after rebase on main Main added HftSessionBase.cacheKey (pipeline-eviction sweeping); the rebased checkpoint commits created sessions without it. * fix(gemini): tighten reactive CachedContent NOT_FOUND matcher isGeminiCachedContentNotFoundError fell through to /NOT_FOUND|not.*found/i on the error message, matching unrelated errors: 'model not found', 'file not found', 'function name not found in declarations', a 404 on an unrelated URL, plain new Error("NOT_FOUND"). When any of these surfaced on a request that referenced cachedContent, the helper deleted the still-valid entry (locally + best-effort server delete) and retried the request inline. If the original error was model-not-found, the retry failed the same way — net effect: 2x latency + billing, and every OTHER consumer of the shared checkpoint id silently paid full re-encode cost on their next call. Rewrite the matcher so a hit requires BOTH: - a structured NOT_FOUND signal (status/code === 404 or "NOT_FOUND", on the top-level error or the nested .error the Google GenAI SDK sometimes wraps GAPI errors in), AND - a scoped mention of the cache resource (cached[_ ]?content) in the message. When only a message signal is available, require the stricter scoped pattern cached[_ ]?content(?:s/[\w-]+)?[^.\n]{0,120}(?:NOT_FOUND|not found|does not exist) — a bare "NOT_FOUND" is deliberately not enough. generateGeminiStreamWithCacheFallback now emits a debug log with {status, code} when a checkpoint-referencing request fails but the error did NOT match a CachedContent NOT_FOUND, so a genuine cache regression stays diagnosable in ops logs. Also expose the two helpers via _testOnly on @workglow/google-gemini/ai so a colocated vitest can drive them without a client / real API call. Tests in packages/test/src/test/ai-provider/Gemini_CachedContentFallback. test.ts cover both true and false cases exhaustively (6 positive, 9 negative), plus four flow tests on the fallback helper: eviction+retry on a scoped NOT_FOUND, and no-op on model-not-found / useCachedContent=false / missing checkpointId. 19 tests, all passing. * fix(test): migrate vi.fn generics to the vitest 4 signature after rebase on main main upgraded to vitest 4, which takes a single function-type parameter instead of the [args] / return pair. Claude-Session: https://claude.ai/code/session_01PwyJuFrJnibKvrrk8Fa4Fn * fix(gemini): let the retry outcome decide whether to evict a CachedContent entry On a recognized CachedContent NOT_FOUND the fallback evicted the shared store entry and only then retried the request inline without the handle. Eviction on suspicion is asymmetric: a false positive destroyed a still-valid CachedContent entry, so every OTHER consumer of that checkpoint silently paid a full re-encode on its next call — while the retry it bought was doomed to fail identically. Decouple "retry without the handle" from "evict the shared entry" and let the evidence decide. The retry is issued first with the entry left in place; only if it succeeds is the handle proven to have been the problem, and only then is it evicted so later requests stop paying the doomed first call. If the retry fails too, the handle was not the problem, so the entry survives for its other consumers. When the retry also fails, the RETRY error propagates rather than the original NOT_FOUND: the cache-free request is the terminal, cache-independent failure and describes the request the caller actually wants to succeed, whereas the original would send them chasing a phantom cache problem. The original's status/code is kept in a debug line so the discarded signal stays diagnosable. A false positive now costs exactly one extra API call and never destroys valid shared state. isGeminiCachedContentNotFoundError stays tight regardless, so a common non-cache error still costs one call rather than two. The tool-use fallback test asserted the old evict-then-retry order and threw a bare "cache not found" 404 that the tightened matcher deliberately does not recognize, so it failed. It now throws a cache-scoped error and asserts the new semantics. Both suites gain coverage for all three outcomes: retry succeeds -> evicted; retry also fails -> entry kept and the retry error propagates; non-cache error -> neither retried nor evicted. Claude-Session: https://claude.ai/code/session_01PwyJuFrJnibKvrrk8Fa4Fn * Add checkpoint session disposal and prefix-rewind KV snapshot support (#706) * fix(ai): core cache-checkpoint review fixes in the task layer - Add model identity to the auto-fingerprint session id so two models sharing a toolset can no longer collide on one local KV session. - AiChatTask: resolve the checkpoint once (memoized) so a sibling task superseding it no longer aborts a live conversation, and pass the new AiSessionContext.seedCheckpointId so providers can attach the checkpoint's warmed state instead of re-encoding the prefix on turn 1. - TextGenerationTask/ToolCallingTask: checkpoint runs return cachePolicy "none" — a cached output would replay a run-scoped checkpoint id whose session no longer exists. - Skip recording empty assistant turns into emitted checkpoint prefixes (Anthropic rejects empty content on replay, poisoning the handle). - finalizeEmittedCheckpoint/CacheCheckpointTask: parent supersede dispose is best-effort — a dispose failure after a successful generation no longer fails the task, and the parent registry entry is always deleted. - CacheCheckpointTask: throw instead of returning an empty-string handle when the checkpoint id is missing. - Export promptToTailMessages from MessageConversion as the single prompt→tail-messages helper for provider checkpoint replay. Claude-Session: https://claude.ai/code/session_0161mGUaqnq2aE5uK2hpWcmq * fix(anthropic,openai,gemini): cache-checkpoint consumer parity and lifecycle fixes Anthropic/OpenAI: - Checkpoint consumers now send prefix.tools (shared toAnthropicTools / merged effective tools on OpenAI), fixing the tool_use-without-tools 400 on replay and restoring warm-up/consumer cache parity — the OpenAI prompt_cache_key now matches the warm-up for tools-warmed prefixes. - annotateLastBlock lifts non-empty string tails into a cache_control text block, so prompt-path emit boundaries actually get a breakpoint. - Emit-only runs (emitCheckpoint with no parent) now write plain-session cache_control breakpoints instead of none at all. - Shared cache_control/tool-mapping helpers exported from Anthropic_CacheCheckpoint and reused by both consumer run-fns. - mergeOpenAICheckpointPrefix keeps the prefix system prompt when the caller passes "" (aligning with the other providers). - buildAnthropicMessages drops empty text blocks / empty-content messages on replay (defense-in-depth against poisoned prefixes). - OpenAI/Gemini private promptTail copies replaced by the shared promptToTailMessages helper. Gemini: - classifyGeminiCacheError now matches the real too-small CachedContent message shape, degrading to inline replay instead of failing the run. - Text-generation consumers refuse tools-warmed CachedContent (the cached declarations would stay active while functionCall parts get dropped). - Cached-content lookup honors seedCheckpointId, so checkpoint-seeded chats reference the warmed cache with tail-only contents. - Proactive stale eviction extracted to one helper used by both run-fns; staleness now checked before tool canonicalization, and the canonical tool key is computed once at warm-up and stored on the entry. Test-only: the reactive NOT_FOUND fallback fake now throws a cachedContent-scoped message, matching the tightened matcher. Claude-Session: https://claude.ai/code/session_0161mGUaqnq2aE5uK2hpWcmq * fix(node-llama-cpp,hft): local-provider cache-checkpoint session fixes node-llama-cpp: - Register a ["session.dispose"] run-fn (worker proxy in worker mode) so checkpoint supersede/scope disposal actually frees worker sessions. - Serialize fingerprint-path session reuse behind withSessionLock and guard reuse on the modelKey, preventing concurrent runs from interleaving one sequence's KV state. - Restore-after-consume: checkpoint sessions rewind via eraseContextTokenRanges back to the snapshot boundary instead of staying consumed after one use (prefix-rewind mode; progressive mode for owned chat sessions keeps per-turn snapshotting). - Checkpoint-seeded chat rebuilds encode the prefix plus full history once instead of dropping prior turns. - The effective system prompt survives setChatHistory rebuilds; the previously masked LocalChatCheckpointSystemPrompt assertion is unmasked. - buildSystemPrompt treats a caller "" as fall-through to the prefix prompt. huggingface-transformers: - Register a ["session.dispose"] run-fn (mirrors the other local provider). - inferHftCapabilities advertises cache.checkpoint, so inferred-capability registrations can actually warm checkpoints. - Session snapshotting extracted to snapshotHftSession, which clones gpu-buffer tensors before the next update() disposes them (WebGPU). - Tools-only prefixes render the chat template with an empty message list instead of throwing. - Chat snapshot reuse keys off the stored encodedText (with an empty-cache fallback) instead of assuming checkpoint parity. - Chats seed from seedCheckpointId when the owned session has no state yet. - The checkpoint text-generation path honors a caller systemPrompt over the prefix's. - mapHFTTools declares its return type. Claude-Session: https://claude.ai/code/session_0161mGUaqnq2aE5uK2hpWcmq * chore: trigger CI (no CI runs recorded on this PR) --------- Co-authored-by: Claude <noreply@anthropic.com> * feat(providers): fill the Usage seam per provider (#624 Phase 2) (#685) * feat(ai): add shared provider usage normalization helpers Adds the mapping layer every provider fills the Phase 1 `Usage` seam through, plus the contract that holds them to one rule. `UsageMapping.ts` normalizes the two OpenAI wire shapes — chat completions (`prompt_tokens`/`completion_tokens`) and Responses (`input_tokens`/`output_tokens`, cache reads and writes nested under `input_tokens_details`) — and exposes the primitives provider packages compose their own quirks from: `toUsageCount`, `usageOrUndefined`, `usageExtra`, and the `OPENAI_STREAM_USAGE_OPTIONS` request fragment. The governing rule is that `undefined` means "the provider did not report this", never "zero": an absent, null, NaN or non-numeric wire field stays `undefined`, while a genuinely reported `0` survives as `0`. Collapsing the two would silently understate spend, since a model that billed no cached tokens and a model that says nothing about caching are different facts. Both shared stream accumulators now return the usage they read off the provider's own terminal frame instead of discarding it: - `accumulateOpenAIResponsesStream` handles `response.completed` (and the `incomplete`/`failed` terminals, which still consumed tokens) rather than dropping them into the `default:` arm. - `accumulateOpenAIChatStream` reads `chunk.usage` BEFORE skipping a choice-less chunk — `include_usage` delivers usage on a final frame whose `choices` array is empty. It takes an optional mapper so a provider reporting extra counters can supply its own. Callers attach the result to their `finish` event as a sibling of `data`, never inside it. The shared contract assertion feeds each provider's mapper the same frames: no payload maps to `undefined`, unreported counters come back `undefined` rather than `0`, a real `0` survives, and only numbers and `undefined` reach the normalized shape. Claude-Session: https://claude.ai/code/session_01PwyJuFrJnibKvrrk8Fa4Fn * feat(anthropic): report token usage from the Messages event stream Anthropic splits its accounting across two frames — `message_start` carries the prompt side (`input_tokens`, `cache_read_input_tokens`, `cache_creation_input_tokens`) while `message_delta` carries the running completion side — and both restate CUMULATIVE figures. `createAnthropicUsageCollector` keeps the newest number stated per field rather than summing restatements, so it never derives a count of its own. A `null` on `message_delta` means "not restated here" (older API versions null the prompt-side fields there), so it leaves the earlier value intact instead of erasing it; a field no frame ever reported stays `undefined`. Mapping: `input_tokens` → input, `output_tokens` → output, `cache_read_input_tokens` → cached, `cache_creation_input_tokens` → cacheWrite, `output_tokens_details.thinking_tokens` → reasoning. `total` is deliberately left unreported — Anthropic states none, and synthesizing one from the parts would misreport cache-discounted input as if it had been billed in full. Wired into TextGeneration, ToolCalling, StructuredGeneration, TextSummary and TextRewriter, all of which already iterate the raw SDK event stream. Claude-Session: https://claude.ai/code/session_01PwyJuFrJnibKvrrk8Fa4Fn * feat(openai): report token usage from the Responses stream and embeddings The Responses API reports usage unprompted on its terminal `response.completed` event, which the shared accumulator was discarding. Reading it there covers TextGeneration, ToolCalling, Rewriter and Summary at once; StructuredGeneration runs its own loop and reads the same terminal events directly. Mapping: `input_tokens` → input, `input_tokens_details.cached_tokens` → cached, `input_tokens_details.cache_write_tokens` → cacheWrite, `output_tokens` → output, `output_tokens_details.reasoning_tokens` → reasoning, `total_tokens` → total. `OpenAI_TextEmbedding` is non-streaming: it maps `usage.prompt_tokens` → input and `usage.total_tokens` → total. The embeddings endpoint reports no completion, cache or reasoning counters, so those stay unreported rather than being zeroed. Claude-Session: https://claude.ai/code/session_01PwyJuFrJnibKvrrk8Fa4Fn * feat(gemini): report token usage from usageMetadata Gemini attaches `usageMetadata` to streamed chunks, restating cumulative counts as generation proceeds. Each run-fn keeps the last non-null block and maps it once at finish, so nothing is summed locally. Mapping: `promptTokenCount` → input, `candidatesTokenCount` → output, `cachedContentTokenCount` → cached, `thoughtsTokenCount` → reasoning, `totalTokenCount` → total. `promptTokenCount` already includes the cached-content tokens that `cachedContentTokenCount` breaks out, matching the input/cached relationship the other providers use. Wired into TextGeneration, ToolCalling, StructuredGeneration, TextSummary and TextRewriter. Claude-Session: https://claude.ai/code/session_01PwyJuFrJnibKvrrk8Fa4Fn * feat(ollama): report token usage from the terminal done chunk Ollama reports counts only on the final `done: true` chunk of a chat stream: `prompt_eval_count` is the prompt side and `eval_count` the generated side (field names verified against the installed SDK's `ChatResponse`/`GenerateResponse` types, not assumed). Intermediate chunks omit both, so `mapOllamaUsage` returns `undefined` for anything but the terminal chunk rather than mistaking an absent count for zero. Ollama runs locally and reports no cache, reasoning or total counters, so those slots stay unreported. Wired into TextGeneration, ToolCalling, StructuredGeneration, TextSummary and TextRewriter. Claude-Session: https://claude.ai/code/session_01PwyJuFrJnibKvrrk8Fa4Fn * feat(providers): opt into include_usage on the OpenAI-compatible providers xAI, DeepSeek, OpenRouter and llama.cpp-server send no usage at all unless the request asks for it, so each streaming chat request now carries `stream_options: { include_usage: true }`. That opt-in appends a final chunk whose `choices` array is EMPTY, which is exactly the shape the `*_StreamChunkSafety` suites exist to guard. Every delta read already optional-chains `chunk.choices?.[0]`, and usage is read before the choice-less chunk is skipped. Standard mapping for all four: `prompt_tokens` → input, `completion_tokens` → output, `prompt_tokens_details.cached_tokens` → cached, `completion_tokens_details.reasoning_tokens` → reasoning, `total_tokens` → total. Two providers report more than that common set, kept in their own packages rather than leaking into the shared helper: - DeepSeek splits the prompt into `prompt_cache_hit_tokens` (the cache-read figure, so it fills `cached`) and `prompt_cache_miss_tokens` (no normalized slot, carried in `extra` — it is what makes DeepSeek's cache-miss-priced cost reconstructable). - OpenRouter reports `cost`, the credits actually charged. That is a price rather than a token count, so it rides in `extra`; it is worth carrying because OpenRouter routes to whichever upstream is cheapest, which the caller cannot reconstruct locally. llama.cpp-server parses SSE itself, so `IChatCompletionDelta` gains a `usage` field and the parser admits a usage-only frame on its own merit — it previously required a content delta, tool calls or a finish reason, which silently dropped the usage chunk. Hugging Face Inference gets no `include_usage` opt-in (it routes to third-party providers whose support varies) but now forwards usage when the upstream volunteers it, instead of discarding the accumulator's return value. Claude-Session: https://claude.ai/code/session_01PwyJuFrJnibKvrrk8Fa4Fn * docs(providers): note why the local providers report no usage node-llama-cpp, HuggingFace Transformers and Chrome built-in AI surface no token accounting: the first two run the model in-process, and the Chrome API exposes only quota measures (`inputUsage`), not billing counters. They therefore leave `usage` absent. Comments record why, so the next reader does not "fix" it by emitting zeros or by counting our own emitted tokens — a local estimate dressed up as a provider fact. Claude-Session: https://claude.ai/code/session_01PwyJuFrJnibKvrrk8Fa4Fn * test(ai-provider): cover usage mapping across every provider Extends the existing per-provider stream-shape suites rather than adding parallel ones, since the terminal `choices: []` usage frame is precisely the chunk shape those suites already guard against. - Xai / OpenRouter `StreamChunkSafety`: assert the request carries `stream_options`, that the terminal frame maps onto `finish.usage`, that OpenRouter's `cost` lands in `extra`, and that a usage frame trailing a structured stream still assembles its object. - `LlamaCppServer_TextGenerationStream`: drives the bespoke SSE parser with a usage-only frame — the regression that motivated admitting a choice-less chunk — and checks it is not mistaken for content. - `Ollama_TextGenerationStream`: the `done: true` chunk's counts, an absent-usage stream, and that counts on non-terminal chunks are ignored. - `DeepSeek_ToolCalling`: the cache hit/miss split end to end. - `Anthropic_SamplingParams`: the two-frame collector — cumulative restatement, `null` treated as "not restated" rather than an erasure, thinking tokens as reasoning. - `OpenAIShapedResponses`: usage off `response.completed`, plus the `incomplete`/`failed` terminals that still consumed tokens. `ProviderUsageNormalization` runs every provider's mapper through the shared contract, so the zero-vs-absent rule is enforced in one place instead of being restated per provider. Claude-Session: https://claude.ai/code/session_01PwyJuFrJnibKvrrk8Fa4Fn * fix(node-llama-cpp): share llamaCppSessions across ./ai and ./ai-runtime bundles The provider ships ./ai and ./ai-runtime as separate dist entry points (dist/ai.js, dist/ai-runtime.js). Both independently import LlamaCpp_Runtime.ts, so its module-level `llamaCppSessions = new Map()` was evaluated twice in-process, producing two distinct Map instances under inline (non-worker) usage. LlamaCppQueuedProvider.disposeSession's local-delete fallback (bundled into ./ai) mutated its own copy while registerLlamaCppInline/setLlamaCppSession (bundled into ./ai-runtime) wrote to the other, so the fallback silently no-opped instead of disposing the session and freeing its LlamaContext sequence slot. Keyed the map behind a Symbol.for(...) on globalThis, the same pattern already used by PortCodecRegistry and the DI Container for state shared across split entry points. Caught by LlamaCpp_SessionDispose.test.ts (added in #706); CI never ran on that PR due to a GitHub Actions outage, so this surfaced only when run locally as part of merging #685 on top of #706. --------- Co-authored-by: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>

Rebase onto current main hit conflicts in files also touched by #684 (queue-adapter deletions), #685/#686 (Usage seam, TaskInvalidInputError), and #641 (AiSessionContext). Per this PR's own conflict-resolution guidance: took main's side on every conflict, then re-ran `bun run format` to reapply the type-import conversion the autofix commit originally made to those files.

…e autofix (#683) * chore(eslint): enforce consistent-type-imports Adds @typescript-eslint/consistent-type-imports so type-only imports are written as `import type`. `disallowTypeAnnotations` is left off because inline `import()` type annotations are the established way optional peer dependencies are typed here without a static import. Claude-Session: https://claude.ai/code/session_01PwyJuFrJnibKvrrk8Fa4Fn * chore: apply consistent-type-imports autofix Mechanical output of `bun run format` (eslint --fix + prettier) after enabling @typescript-eslint/consistent-type-imports. No hand edits. Claude-Session: https://claude.ai/code/session_01PwyJuFrJnibKvrrk8Fa4Fn * chore: convert the remaining inline type specifiers to top-level import type Claude-Session: https://claude.ai/code/session_01PwyJuFrJnibKvrrk8Fa4Fn * chore: reapply consistent-type-imports autofix after rebase conflicts Rebase onto current main hit conflicts in files also touched by #684 (queue-adapter deletions), #685/#686 (Usage seam, TaskInvalidInputError), and #641 (AiSessionContext). Per this PR's own conflict-resolution guidance: took main's side on every conflict, then re-ran `bun run format` to reapply the type-import conversion the autofix commit originally made to those files. --------- Co-authored-by: Claude <noreply@anthropic.com>

sroussey force-pushed the ai-provider-cache-checkpoints branch from 34f99ac to 5fdabf6 Compare August 6, 2026 18:30

sroussey force-pushed the claude/libs-issues-triage-prs-mh6x2o-624-phase2 branch from acfcfd4 to 2734473 Compare August 6, 2026 19:55

sroussey and others added 9 commits August 6, 2026 22:14

sroussey force-pushed the claude/libs-issues-triage-prs-mh6x2o-624-phase2 branch from 2734473 to 62fb430 Compare August 6, 2026 22:23

sroussey merged commit acd52b7 into ai-provider-cache-checkpoints Aug 6, 2026

sroussey mentioned this pull request Aug 7, 2026

perf: migrate tool-calling argument parsing to createPartialJsonStream #709

Closed

sroussey deleted the claude/libs-issues-triage-prs-mh6x2o-624-phase2 branch August 13, 2026 05:02

sroussey mentioned this pull request Aug 14, 2026

Add uniform usage/token telemetry across all AI providers #624

Closed

12 tasks

Navigation Menu

Uh oh!

feat(providers): fill the Usage seam per provider (#624 Phase 2) - #685

feat(providers): fill the Usage seam per provider (#624 Phase 2)#685
sroussey merged 9 commits into
ai-provider-cache-checkpointsfrom
claude/libs-issues-triage-prs-mh6x2o-624-phase2

sroussey commented Aug 5, 2026

Uh oh!

sroussey commented Aug 5, 2026

Uh oh!

github-actions Bot commented Aug 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

sroussey commented Aug 5, 2026

Stacked PR — merge after #641

The undefined vs 0 rule

Per-provider mapping

Providers that report nothing — usage stays absent

Notes on specific decisions

Shared plumbing

include_usage safety

Tests

Verification

Inherited failure — not from this PR

Uh oh!

sroussey commented Aug 5, 2026

Uh oh!

github-actions Bot commented Aug 5, 2026

Coverage Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

The `undefined` vs `0` rule

Providers that report nothing — `usage` stays absent

`include_usage` safety