tito: per-turn token record persistence + abridge composition (production capture)

[Meirtz]

Makes the TITO gateway's token capture production-ready end to end: a byte-fidelity verdict on the span capture path (P0), durable per-turn token records + session lifecycle on the gateway (P1), and the abridge serve-mode composition that puts an Anthropic-speaking agent in front of the gateway with joinable message-level capture (P2). A follow-up adversarial review (18 confirmed findings, 5 blockers) is fully addressed in the last two commits — see "Review response" below.

P0 — byte-fidelity verdict

A downstream consumer's captured-span fixture showed every gen_ai.prompt.*.content cut at exactly 120 chars, mid-word. Verdict from source: agentix does not truncate anywhere on the capture path. populate_*_span stores full strings, Span.set_attribute keeps raw values, JsonlProcessor writes Span.export() verbatim, the SIO trace bridge forwards full attrs, and the OTel exporter sets no value-length limit (the OTel Python SDK default is unlimited). The fixture was hand-trimmed. Two deterministic tests pin the property (>2000-char prompt survives populate → processor → JSONL round-trip byte-for-byte, and the OTel export leg untruncated).

The record contract — tito.record.v1 (normative)

The gateway is the producing, normative side of this contract; the schema document lives in plugins/tito/README.md and downstream consumers adapt to it. One flat JSON line per committed turn, flushed per line, in <record_dir>/<session_id>.jsonl:

{
  "schema_version": "tito.record.v1",
  "session_id": "<gateway session id>",
  "thread_id": "<x-thread-id echo>",        // OPTIONAL: omitted when header absent
  "request_id": "<x-request-id echo | null>",
  "turn_index": 0,                           // monotonic; failed lines leave a detectable gap
  "ts": 1750000000.0,
  "model": "<chat request model | null>",
  "backend_kind": "sglang" | "vllm",
  "sampling": { "temperature": 0.6, ... },   // whitelist lifted from the request body; {} = backend defaults
  "tokenizer": { "checkpoint": "...", "tokenizer_sha256": "<64 hex>", "chat_template_sha256": "<64 hex>" },
  "prompt_token_ids": [ ... ],
  "prompt_segments": [ {"start": 0, "end": 42, "source": "prefix"}, ... ],
  "completion_token_ids": [ ... ],
  "completion_logprobs": [ ... ],            // finite floats, 1:1 with completion ids, same forward pass
  "assistant_message": { ... },
  "finish_reason": "stop" | "tool_calls" | ... | null,
  "prefix_stable": true,
  "render_skew": null | {"equal": false, "first_divergence": 17}
}

Key semantics:

• tokenizer.tokenizer_sha256 — SHA-256 over the tokenizer definition bytes: fast tokenizers hash the complete serialized tokenizer.json (backend_tokenizer.to_str()); slow tokenizers the sorted-JSON vocab. Equal value ⇔ identical tokenization. chat_template_sha256 hashes the template actually in effect.
• prompt_segments.source — gateway-native, exhaustive vocabulary: render (from-scratch render: first turn or prefix-window-outrun fallback), prefix (reused accumulated checkpoint), system/user/tool (per appended role), generation_prompt. Segments tile the prompt exactly — loss-mask material with no downstream re-tokenization.
• prefix_stable — computed by the sink against the last line it actually wrote (not the in-memory checkpoint): true iff this prompt extends the previous recorded line's prompt+completion. An applied rollback whose turn never produced a line (upstream error/timeout/409) surfaces as false on the next recorded turn. false ⇒ do not splice into one linear token stream.
• Strictness — lines are strict JSON (allow_nan=False); non-int token ids / non-finite logprobs are rejected at the sink (logged + turn_index gap), never coerced. A closing tito.session.v1 line (reason: deleted/ttl_evicted/capacity_evicted/shutdown) finalizes each file.

P1 — gateway persistence + lifecycle

--record-dir (env TITO_RECORD_DIR); no record dir → in-memory behavior unchanged. vLLM logprobs are retained through TurnHarvest (previously validated-then-discarded); both backends record sampled-token logprobs from the same forward pass. Lifecycle: harvest-then-DELETE remains the documented flow; optional --session-ttl-seconds / --max-sessions evict idle/LRU sessions — always flushing + finalizing the record file first, never touching an in-flight session, dropping the pool pin like DELETE. The idle clock starts when a turn ends, so a generation slower than the TTL never lets the agent's next request evict its own session. Interleaved turns on one session are an explicit 409 (ConcurrentSessionUpdateError) instead of a silently unrecorded response.

P2 — abridge composition + recorder alignment

• agentix-bridge-serve --tito-url (mutually exclusive with --upstream-base-url): Anthropic shell over the token-recording gateway. Because real Anthropic agents multiplex conversations over one key (Claude Code helper calls, Task subagents, reruns) while a gateway session is one linear history, each caller session demuxes by conversation: key = canonicalized (system, first user message), one AnthropicToOpenAI(SessionForward) — one gateway session — per conversation. Documented boundaries: deep compaction keeping the opening rides the gateway's rollback/from-scratch paths (past-window rewrites are its documented 400); byte-identical openings collide; caller-LRU eviction mid-rollout splits gateway-side capture (size --max-sessions above the concurrent key count). --tito-delete-on-evict (default off) reaps gateway sessions when a caller session closes.
• --record-dir (env ABRIDGE_RECORD_DIR, either mode): per-session Recorder rows carry session_id, request_id, and (tito mode) gateway_session_id — the gateway's own session id, restoring the session-level join; request_id is bound on a context var and reaches the upstream as x-request-id, so message rows join token records per call. Recorder is log-and-serve (capture failure never fails the agent's call) and never resurrects a closed file.
• Translation-gap status: tool_choice passes through (auto→auto, any→required, none→none, {type:tool,name}→named function; four xfails green; disable_parallel_tool_use stays dropped, matching the reference translator). Assistant thinking history is forwarded as reasoning_content (vLLM/sglang dialect; the Anthropic crypto signature dropped) — including thinking-only assistant turns; the fixture xfail stays, re-documented as a deliberate dialect divergence from the LiteLLM golden (thinking_blocks), with positive contract tests. Request-level thinking budget, images, long tool names, metadata.user_id, top_k, stop_sequences remain documented xfails.
• Translator version pin: TRANSLATION_SPEC_SHA = one SHA-256 over the transforms module and both client modules that shape the upstream body (assistant replay, forced stream=False, model override, upstream_params), reported on GET /_health.

Review response (18 confirmed findings)

Blockers

• prefix_stable vs unrecorded rollback (repro'd) — fixed: sink-owned stability baseline = last recorded line; reviewer repro regressed as test_unrecorded_rollback_turn_still_breaks_prefix_stability.
• Flat vs nested record shape; missing policy/tokenizer_sha256; missing sampling; segment-source vocabulary — resolved by contract ruling: the producer is normative (schema section above / README); added sampling, the tokenizer block with a genuinely computed tokenizer_sha256, and thread_id passthrough; structure stays flat; segment sources stay gateway-native. Consumers adapt.

Should-fix

• TTL counts generation as idle (repro'd) — fixed (idle clock starts at turn end); reviewer repro regressed as test_generation_time_does_not_count_as_idle_for_ttl.
• Sink except OSError only + skipped index increment — fixed: except Exception, increment in finally (any failure leaves a detectable gap), strict-JSON validation at the sink.
• One-key-one-gateway-session bricks multiplexing (both lenses) — fixed by the conversation demux (+ regression test).
• No gateway-session lifecycle closure on caller eviction — --tito-delete-on-evict added; the eviction-split residual is explicitly documented (README + serve docstring).
• Spec sha covered one module — fixed: joint hash over the three body-shaping modules.
• thread_id never emitted — fixed: x-thread-id passthrough into records.

Nits

• Recorder _write unprotected — fixed: log-and-serve, aligned with the gateway sink policy.
• Recorder reopens file after aclose — fixed: closed-guard drops late rows loudly.
• Thinking-only assistant history dropped — fixed: passes the append guard with reasoning_content.
• session_id_for docstring / join-key distortion in tito mode — fixed: gateway_session_id row field + docstring caveat.
• NaN / numpy scalars via default=repr+allow_nan — fixed: allow_nan=False + sink rejection of non-int ids / non-finite logprobs.

Test coverage

P0 fidelity round-trips; tito contract over the real ASGI app (record shape incl. sampling/tokenizer/thread_id, logprobs 1:1, per-line crash-safety, prefix_stable true/false incl. the unrecorded-rollback break, interleave 409, TTL/capacity eviction flush + in-flight immunity + generation-time immunity, strict-JSON gap detection, shutdown finalize, vLLM render_skew/logprobs/finish_reason); a real-Qwen3-tokenizer golden (network-marked, cache-or-skip offline) proving incremental == from-scratch token-exactly with segment boundaries decoding to role markers; abridge composition e2e over a real-HTTP fake gateway (per-conversation demux, delete-on-evict reap, identity headers, SSE replay, row↔record joins incl. gateway_session_id), Recorder id/robustness suite, tool_choice fixtures green, /_health joint sha pin.

Gates (at 13c25f2)

• uv run ruff check . → All checks passed!
• uv run pyright → 0 errors, 0 warnings, 0 informations
• uv run pytest tests/ plugins/ -q → 757 passed, 1 skipped, 6 deselected, 18 xfailed
• uv lock --check → clean

🤖 Generated with Claude Code