feat(models): prompt-cache observability + an unconditional cacheable prefix on every Anthropic request - #49
Merged
rezaho merged 13 commits intoAug 2, 2026
Conversation
…R-011) Two additive, optional, keyword-only params on Orchestra.resume_session so a consumer that did not execute() the run in this process can resume a snapshot cleanly: - canonical_topology: binds topology_graph internally via a new private _build_topology_graph (the analyze + legacy-shim + validate core extracted from execute(); the single source, so the resumed graph is equivalent to execute's). Bound before the RESUME_NO_TOPOLOGY guard + digest check. - on_bus_rebuilt: invoked once after the resume bus rebuild AND after the topology/digest preconditions pass (never on a failed-precondition resume), letting the consumer re-attach custom EventBus subscribers the rebuild drops. Both default to today's behavior; execute() refactored to call the helper (behavior-preserving). 6 new tests; full pause/resume suite green.
…FW17/ADR-011) A deterministic completing-resume integration test: a real on-disk paused snapshot (gated-runtime + pause_session) resumed via resume_session(canonical_topology=) with a stub Agent (overrides _run -> return_final_response, no LLM) registered in the AgentRegistry. Verifies the resume binds the topology, drives the real agent through RealRuntime to a terminal OrchestraResult, and that a subscriber attached inside on_bus_rebuilt receives a BranchCompletedEvent from the resumed dispatch — closing the AC-1 terminal + AC-4 real-event coverage the empty-state tests left open.
…tes FW17) resume_session rebuilds self.event_bus and _wire_event_bus re-creates the listener set (TraceCollector / StatusManager / AGGUITranslator) on it, but the REUSED step_executor (which emits LLMCallEvent — the event per-run cost is computed from) and _user_node_handler still held the prior bus. So a consumer re-attached via resume_session(on_bus_rebuilt=...) — FW17's motivating per-run cost adapter — received no events from the resumed dispatch, and post-resume LLM spend went unbilled. FW17's own resume test missed it (stub agent makes no LLM call; on_bus_rebuilt was only exercised against orchestrator-level events). A real-model pause->resume surfaced it. Fix: _wire_event_bus also re-points step_executor.event_bus and _user_node_handler.event_bus to self.event_bus, guarded (they are created after this call in __init__, so the guards no-op there and bind the fresh bus at construction; on resume they exist and get re-pointed). Framework pause/resume suite (34) + full suite (1295 passed, same 3 pre-existing failures) green. ADR-011 completion note + CHANGELOG updated.
…DR-012) Make the ask_user/UserNode wait durable: a workflow reaching a durable user interaction snapshots to disk and Orchestra.execute() returns paused-awaiting-user (metadata[paused]+[awaiting_user]; WorkflowResult.error="awaiting_user", a second pause sentinel) WITHOUT an in-memory wait or the 300s timeout. After an arbitrary wait and/or a full restart, resume_session(session_id, user_response=...) restores the suspended branch, injects the response via the existing resume_branch_with_user_response seam, and continues to terminal. SYNC ask_user is unchanged (durable=False default; the durable path bypasses _drive/handler entirely). - pending_user_interaction scalar, held apart from the FIFO sibling deque (which would mis-dispatch it); a 5-tuple live / Optional[UserInteractionState] on-disk so durable siblings serialize FIFO durably rather than silently downgrading to SYNC. - _snapshot_and_write extracted from pause_session; execute()/resume_session self-write on the awaiting-user exit (inside the try, before the finally pop). - resume_session gains user_response (additive, keyword-only, rebased onto FW17's canonical_topology/on_bus_rebuilt) + AC-5 arg-validation. - durable flag on enqueue_user_interaction (impl + DetNodeContext protocol) and UserNode, declarable in a workflow spec via a USER node's metadata[durable] (legacy shim read, forward-compatible to the v0.4 generic det-node path). - Fix a latent SYNC-seam gap: resume_branch_with_user_response now de-candidates the terminated suspended branch (via _unregister), so its barrier can fire. Surfaced by the first orchestrator-level UserNode->resume->terminal test (Session 03 had none). ADR-012 (approved) + CHANGELOG + 19 deterministic tests + 1 gated live OAuth test. Framework suite: 1459 passed, 52 skipped, same 4 pre-existing failures + 2 readline collection errors (no new failures).
…DR-013) A granted agent emits escalate_to_user(prompt) to durably suspend a run for human input WITHOUT a topology User node — the dynamic counterpart to ask_user. Scoped via a per-agent can_escalate grant (default off) gating BOTH the schema offer and validation; a new ESCALATE_USER step-kind routes through the orchestrator's _interpret directly into framework 16's durable seam (enqueue_user_interaction(..., durable=True)) with resume_agent = the emitting agent, so resume_session(user_response=...) re-runs that agent. The FW16 durable suspend/resume machinery is reused unchanged; the deferred general control-directive primitive (pause/redirect/fail) is left unbuilt. 26 deterministic tests + 2 gated live-OAuth tests (a real model emits the directive and resumes to terminal). Full framework suite 1491 passed / 54 skipped; the 3 failures + 2 readline collection errors are pre-existing (Windows/worktree), verified on the baseline.
…ng-user prompt (S62) Completes FW18/ADR-013's wire-mirror and result-report so a consumer (Spren S62 browser re-auth) can grant + surface escalation through a serialized workflow: - AgentSpec.can_escalate round-trips AgentSpec<->Agent, mirroring bidirectional_peers across the field, agent_to_pydantic (live->spec), and both pydantic_to_agents ctor paths. A workflow agent hydrated from a spec can now carry the grant. Additive, backward-compatible (optional, default False; absent key tolerated under extra=forbid). - Orchestra.execute and resume_session now carry the pending interaction's prompt on the awaiting-user result metadata (awaiting_user_prompt, both exits), so a consumer renders the re-auth reason without parsing the snapshot. Round-trip tests added (tests/agents/test_serialize.py); ADR-013 wire-mirror addendum + CHANGELOG entry. Framework stays Spren-agnostic (SP-018).
…r tool-search / defer_loading)
A per-tool `defer_loading: true` flag (top-level on the tool dict) marks a tool for on-demand
discovery: the provider's tool-search built-in finds it when needed and the definition rides the
message tail, so the deferred tool's schema stays out of the billed/cached request prefix and the
prompt cache survives a mid-conversation load. The flag rides the existing `tools` array — no
arun/run signature change (a separate kwarg would be warn-dropped by OpenAI's allowlist).
Per-provider translation (DP-006):
- Anthropic (api-key + OAuth): maps defer_loading onto the Anthropic tool + auto-adds the
tool_search_tool_regex_20251119 server tool.
- OpenAI Responses (api-key + OAuth/Codex): maps it onto the flat function tool + auto-adds
the {"type":"tool_search"} built-in.
- OpenRouter: strips the flag before its verbatim forward (+ warns) so it can't 400 the wire.
- Google: warns + eager fallback (its rebuild already drops the flag).
- local.py: untouched (separate base, tools unsupported).
Additive / zero-regression: with nothing deferred every adapter's payload is byte-identical to
before (no defer_loading key, no search tool, no header change — the Tool Search Tool is GA, no
anthropic-beta header needed). Response side unchanged: the discovered tool's tool_use already
surfaces through every harmonizer and the API re-expands tool_reference from the tools= defs, so
the search->load->use round-trip is coherent without carrying the server-search blocks (verified
live on OAuth). Tests: tests/models/test_deferred_tool_loading.py (per-adapter native shape, the
nothing-deferred identity guarantee, openrouter-strip/google-warn leak-prevention, and the
cache-prefix-stable-across-a-discovery-round-trip proof).
… error A model that runs to natural completion and produces no content blocks is returning a success: "I have nothing to say." Both Anthropic adapters raised a typed ModelAPIError on it instead, and the classifier pinned that error non-retryable — so every silent turn died terminally. This is a behaviour callers explicitly ask for. An agent instructed to stay quiet when it has nothing to report (a heartbeat that found no work, an inbox check that found no mail) obeys, ends the turn with zero content blocks and stop_reason 'end_turn', and the provider bills it as a successful response. Confirmed on the wire: message_start (content []) -> message_delta (stop_reason end_turn, output_tokens 2) -> message_stop, no content_block_start. The raise was a symptom. The root is that HarmonizedResponse could not represent the outcome: its validator rejects content=None with no tool_calls, the adapters normalize empty text to None, and the empty-output branch had only two arms (deterministic truncation -> placeholder; everything else -> raise). The typed error was added to replace an UNKNOWN ValidationError, which improved the error's quality while cementing a success as an error class. The validator already blesses the shape this needs: it checks `content is None` specifically, because an empty string is a valid response from some providers. The API-key adapter already uses that escape for thinking-only responses. So the empty-output branch gains its correct third arm — end_turn harmonizes to content="" — beside the two that exist, rather than a parallel path. Refusal and a stream that closed with no terminal at all keep raising: those are genuine anomalies, and that distinction is the value the typed-error work added. The test asserting empty end_turn raises is INVERTED, not removed: it encoded the wrong contract. Its replacement pins the silent turn, and a twin covers the API-key adapter so one provider keeps one behaviour. The now-unreachable end_turn classification arm is deleted.
A connect/DNS/timeout/reset error raised by the HTTP client (httpx, httpcore, aiohttp, or stdlib) carries no HTTP status and no provider error body, so it fell through every branch of ModelAPIError.from_provider_response and kept the UNKNOWN, non-retryable default. The dead 'mark timeout/network as retryable' block never fired because nothing ever assigned NETWORK_ERROR. Consequence: on a network-flaky host a transient blip (e.g. a DNS hiccup, '[Errno 11001] getaddrinfo failed') was classified terminal, so the turn was permanently dropped instead of retried by the existing backoff ladder. Rate limits and 5xx retried; a dropped packet did not — inverted. Classify statusless transport exceptions by walking the raised exception's MRO by class name (one table across all HTTP libs, no new imports — httpx is not a declared dep). Timeouts -> TIMEOUT, connect/ network/reset -> NETWORK_ERROR, both flowing through the pre-existing retryable block. Fires only when status_code is None and we are still UNKNOWN, so it never overrides a real provider verdict. Our-side/config transport faults (LocalProtocolError, UnsupportedProtocol) are deliberately excluded and stay non-retryable. Inverts test_true_connection_error_no_status (it asserted the buggy 'stays unknown'); message text is still preserved verbatim.
Claude Opus 5 and Sonnet 5 reject `temperature` and a fixed
`thinking.budget_tokens`. Both are hard 400s, not ignored fields, so the
previous payload made these models unusable on their first turn. Opus 4.7/4.8
reject `temperature` too, and the OAuth adapter set it unconditionally — that
leg was already failing for them before the Claude 5 line existed.
Capability now keys off the model, not the spelling: one predicate pair behind
a normalizer that collapses the `anthropic/`, `anthropic.` and `us.anthropic.`
prefixes, so the same model resolves identically whether it arrives from the
first-party API, OpenRouter, or Bedrock. Thinking becomes `{type: "adaptive"}`
for that family, with depth steered by `output_config.effort` (a positive
budget keeps its "thinking on" meaning). `ModelConfig` also accepted only
`minimal..high`, which rejected `xhigh`/`max` at construction and left the
effort path unreachable for the two tiers that matter most on these models.
Add a `bedrock` provider for Claude on Amazon Bedrock. The
`bedrock-mantle.<region>.api.aws` endpoint speaks the standard Messages API and
returns real SSE, so this is a thin subclass of the Anthropic adapter rather
than a parallel one — message conversion, tool handling, stream accumulation
and harmonization are all inherited. Its deltas are bearer auth, the mandatory
`anthropic.` id prefix, and no `output_config.format`/`strict` support, which
degrades a schema request to the prompt fallback carrying the real schema
instead of putting an illegal field on the wire.
`metadata.model` is what cost meters price on, and Bedrock echoes a *bare* id
for a request made with a prefixed one — trusting the echo prices the whole
provider at zero with no warning at the meter. `report_model_id` keeps the
requested spelling where the echo would not round-trip, while first-party
adapters still prefer the echo so an alias resolves to its concrete snapshot.
Every shape here was measured against the live endpoints, both legs.
…cache usage is observable Nothing in this codebase placed a prompt-cache marker over conversation content, so every call re-paid full input price on the entire conversation. Measured before the change on a four-step turn: 4,263 input tokens, zero cache reads. After: 8 uncached tokens and 4,162 read from cache — $0.0230 to $0.0051 on that shape. The marker goes on the last content block of the last message, on every request — the platform's multi-turn caching pattern. A breakpoint reads any entry written at or before it, so marking the growing tail each time both reads the previous request's entry and extends it. It also satisfies the 20-block lookback by construction: a per-request tail marker is always a handful of blocks behind the last one, whereas a marker placed once silently stops matching in an agentic turn that appends several blocks per step. Adapter-owned and unconditional rather than a caller opt-in. Caching is prefix-match arithmetic over the rendered payload, and only the payload builder knows where block boundaries land — a caller cannot place this correctly even if it wanted to, and a caller that forgets silently pays full price forever. This matches the one existing cache_control site (the OAuth adapter's static prefix block) and `defer_loading`, the nearest analogous feature, which deliberately took no new request parameter either. The accepted cost is a ~25% write premium on a large genuinely-one-shot call; a prompt under the model's cacheable minimum writes nothing and costs nothing. The api-key leg's `system` moves from a bare string to the array form. Same bytes to the model, but the string form makes the system tier structurally unable to carry a marker. No marker is placed there yet: on that leg the system content is the caller's per-turn prompt, which for the caller this ships for changes every turn, so a marker would write a fresh entry per call and read none. Observability, because none of the above is verifiable otherwise: `UsageInfo` gains `cache_read_input_tokens` and `cache_creation_input_tokens`, and both Anthropic harmonizers populate them from the raw usage dict they already receive. `total_tokens` keeps its established meaning; the cache-aware reading is the new `full_prompt_tokens` property. `prompt_tokens` alone is the UNCACHED REMAINDER, so a consumer that sizes a conversation, prices a call, or bounds prompt growth on it under-measures by up to ~10x once a cached prefix exists — silently, since nothing errors. Providers reporting no cache figures harmonize to None and are unaffected. The OAuth adapter's hand-rolled stream reader now MERGES usage across events instead of discarding `message_start` and assigning at `message_delta`. Both twins. The observable defect this fixes is narrower than expected — a live pre-change probe returned `prompt_tokens=27`, because that endpoint's `message_delta` carries the whole usage dict — but `message_start` is the only event carrying the cache-TTL breakdown, `service_tier` and `inference_geo`, and a stream ending without a `message_delta` harmonized with no usage at all. Verified live on three endpoints: the write/read pair round-trips on the OAuth leg (1,219 written then read on claude-sonnet-4-6) and on Bedrock (1,564 on anthropic.claude-opus-5), and a non-streaming request reads an entry a streaming request wrote, in both directions.
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every Anthropic request now writes a cacheable prefix, and cache usage is visible on the harmonized response. Measured against the live API (Claude Opus 5 on Bedrock, and the OAuth leg): an ordinary multi-step agent turn reads 13.6K tokens from cache and costs 57% less; a conversation-fold request reads 99% of its prompt.
What changes
UsageInfogainscache_read_input_tokens+cache_creation_input_tokens, plus afull_prompt_tokenssum. This matters beyond reporting:input_tokensis the uncached remainder only, so any consumer treating it as the whole prompt silently under-counts by up to ~10x once caching is active. Both Anthropic harmonizers now populate the fields; every other provider harmonizes toNoneand is unaffected.A
cache_controlmarker on the tail of every Anthropic request (mark_conversation_tail_for_cache), on both the api-key and OAuth legs. Adapter-owned and unconditional rather than a caller opt-in: only the payload builder knows the rendered block layout, and this matches the one pre-existingcache_controlsite. Idempotent, never more than one tail marker, and a request whose last content is a plain string has it promoted to a one-block list.The api-key leg's
systemmoves to array form so it can carry a marker. No marker is placed there yet — that is correct only once the caller's system prompt is byte-stable across turns.The OAuth stream reader now merges usage across events instead of discarding
message_startand assigning atmessage_delta. Latent in most cases (the delta repeats the input-side figures), butmessage_startis the only event carrying the cache-TTL breakdown, and a stream ending without amessage_deltapreviously harmonized with no usage at all.Notes for review
read=16350 write=29, same request minus 6 messagesread=0 write=14544.main.c0b2b6a) is contained here.Tests
tests/models/— 201 passed, 39 skipped, including a newtest_prompt_cache_breakpoint.py(504 lines) covering marker placement, idempotency, the 4-marker ceiling, the string-to-block promotion, and cache-usage harmonization on both legs plus a no-cache provider.Full suite: 1,449 passed, 53 skipped, 4 failed. All 4 failures reproduce identically at
origin/mainin a clean worktree (doc-generation drift, a storage key assertion, a topology constructor check, and a Windows file-lock test) — pre-existing, untouched by this branch.