v0.23.0

[0.23.0] - 2026-08-26

The release where the production obligations land, and the first artifact to
carry them: everything between v0.22.1 and here ships for the first time,
including the [0.22.2] section below, whose tag was never cut. Generations and
checkpoints with hot backup and one-flag restore; erasure with a receipt that
re-checks every domain by name; tenant identity in the primary key; the eval
entity model; attribution that names the failing step; a compute deadline on
every query and a connection slot no panic can leak; and a six-demo tour that
CI holds green. The format pays for it once: segment formats 1–5 are spent,
and a pre-generation store migrates on first open, automatically.

Added

• --query-deadline-ms (default 30000, 0 disables): a wall-clock budget
  one query or aggregation may spend inside the engine, checked at segment
  boundaries and every few thousand decoded records. Past it the request
  answers 400 {"error":"query deadline exceeded: ..."} with no rows — the
  refusal that stops one runaway scan (an unindexable content search, a
  whole-corpus fold) from holding a handler thread for minutes, and never a
  partial aggregate, because a partial answer would look complete. A 200
  still means what it meant — every answer served is still the whole answer
  — and durability is untouched: ingest, the tail, exports (bounded by the
  socket timeout), and erasure work are all exempt. The worst measured
  legitimate query in this repo is ~3 s, so the default never touches
  measured use. traza_query_deadline_exceeded_total counts the HTTP
  refusals; an MCP tool that exhausts the budget surfaces the refusal in its
  tool result instead, uncounted.

  • Library surface: Config::query_deadline is a new field (None —
    the unbounded behaviour every embedder had — by default; a Config
    built by struct literal must add it, ..Config::default() need not),
    and Error::DeadlineExceeded is a new variant of the exhaustive
    Error enum, so a downstream exhaustive match must add an arm.

• A demo tour in examples/ — six scripted proofs, one claim each. Five
  new demos join mcp-demo, and examples/README.md
  indexes the tour: swarm streams a simulated agent platform into a live
  dashboard in real time; crash SIGKILLs the server mid-ingest and proves
  every acknowledged span survived, then corrupts a byte, watches
  /v1/verify name the file, and restores from a hot backup; needle
  ingests a million spans over real HTTP and times the answers; incident
  investigates a runaway session over MCP — diagnose_session, a defused
  prompt injection, promote_failures_to_dataset, and an experiment diff
  proving a fix; vanish erases one tenant's session next to an identical
  twin tenant and walks the receipt, including the backup pin it refuses to
  ignore. Every number each demo prints is measured during the run — a timed
  request, a counter, a field from a live response — and each script asserts
  its own claims and exits non-zero when one fails, which is why ci.sh now
  runs all six at their smoke settings as merge gates.

  • Demos refuse to run against a port another server already holds:
    readiness is the child's own startup line, not a probe that a squatter
    can answer. This guard also landed in mcp-demo.

• Attribution: the server names the failing step, instead of the reader
  doing it. A new diagnose_session MCP tool answers why did this run
  fail — the outcome, the step the failure is attributed to, and the
  repetition behind it — with the evidence for each claim travelling beside
  it. It replaces the two prompts that used to teach a model to read a trace
  and judge its shape by eye ("repeated sibling names are a retry storm,
  deep chains are a loop"), on the surface's own rule that a prompt wanting
  a branch is a tool.

  • Every rule fires on telemetry that already exists. An earlier design
    rested on a session.outcome attribute and a relation: "retry-of" link
    — both conventions Traza invented and nothing else emits — which would
    have been a closed loop: seed the convention, detect it, pass CI, and
    answer cause: null on every real store. A declared convention now
    raises confidence and is never a precondition. The discriminators are
    error density, serial fraction, self-similar depth and context growth.
  • Context growth is read from context_tokens, not prompt tokens.
    gen_ai.usage.cache_read_input_tokens and cache_creation_input_tokens
    are recognized, because Anthropic reports input_tokens as the uncached
    remainder: a growing conversation reports a prompt count that falls as
    its cache warms, so reading the wrong field inverts the signal on the
    configuration long-running agents actually use. Where the arithmetic is
    unknown the trend is reported as unreadable rather than guessed.
  • It refuses to say more than the data supports. A shape is classified
    only when a discriminator agrees; otherwise it is reported as
    inconclusive with the missing signal named. Ordinary iteration is
    reported as ordinary so a reader can see it was examined and set aside.
    Absent token data can never satisfy a healthy classification. The
    strongest test is silence: the analysis finds no fault anywhere in the
    bundled corpus of healthy workloads.
  • Session outcome is first-class and provenance-tagged: declared when
    a span said so (session.outcome, session.goal — Traza extensions,
    labelled as honestly as llm.cost_usd), derived from the run's own
    spans otherwise, and unknown — never rendered as success — for a run
    still in flight. A run that failed a call and recovered is a success that
    reports its errors, not a failure.

• promote_failures_to_dataset closes the loop into the M4 eval
  entities: a session's attributed failures become a regression dataset
  version, each example carrying its own copy plus provenance back to the
  span. Re-promoting is idempotent. Gated by an rw token and a new
  --mcp-promote switch, separate from --mcp-annotations because an
  annotation dies with its span while a promoted example deliberately
  outlives its source. The caller names a session, never a span: the
  promoted set is re-derived server-side from the diagnosis, so text injected
  into the telemetry can change whether a promotion happens but never what it
  copies — proven by a test that plants exactly that instruction.

• A runaway agent scenario in the seed corpus: an agent whose search tool
  fails from the fourth turn on, so every later reflection appends the failure
  to its context and asks again. Its steps are siblings under one root, the
  shape real frameworks emit, and it carries no declared outcome and no retry
  link — so finding it proves the analysis works on what a real pipeline
  sends.

• Tenant identity in the primary key. Span identity is now
  (tenant, trace_id, span_id) — everywhere: the write buffer's index, WAL
  replay, segment supersede resolution, compaction's last-write-wins merge,
  the tail ring's veils, cursors, the analytics key hashes, annotations and
  erasure. Two tenants sharing a trace id can no longer silently upsert over
  each other, which is the whole reason this ships now: keys cannot be
  retrofitted after the format freeze. The default tenant is the empty
  string and is never serialized, so a single-tenant store writes
  byte-identical WAL frames, segment records and annotation lines to what it
  wrote before tenancy existed — no segment or WAL format bump, proven by a
  serialization-guard test. Tenant scoping reaches every surface the roadmap
  names:

  • Credentials: TRAZA_TOKENS entries take an optional binding,
    scope@tenant:token. A bound credential ingests, queries, tails,
    exports, annotates and erases exactly its own tenant; naming another
    tenant is a 403, and the store-global operator surfaces (/v1/stats,
    /v1/metrics, /v1/verify, checkpoint/flush/backups) refuse bound
    tokens outright. OTLP exporters select a tenant with the traza.tenant
    resource attribute; MCP tools are scoped by the same binding.
  • Sessions are (tenant, session_id) — the same session.id under
    two tenants is two sessions, in the rollup sidecar (format v3,
    self-healing rebuild), the session list, and group_by=session rows.
  • Retention takes per-tenant windows: --tenant-ttl TENANT=SECONDS,
    repeatable. A tenant's cutoff is its override, else the global TTL,
    else never — and the segment retire-whole fast path only runs when a
    global TTL covers everyone, because a whole-segment decision taken from
    configured windows alone would delete an unswept tenant's data with the
    segment.
  • Quota accounting: GET /v1/tenants reports per-tenant spans,
    traces, serialized bytes and offloaded payload bytes from one exact
    fold. Accounting, deliberately not enforcement.
  • Erasure: trace/span/session subjects carry a tenant (empty = the
    default tenant, never "all"), and a new tenant subject erases
    everything a tenant owns — spans, annotations, scores, datasets,
    experiments — with the same barrier, ordering, and receipt discipline
    as M3. Reference-aware payload deletion now spans tenants: a blob two
    tenants share survives one tenant's erasure and the receipt names why.

• The eval entity model — identity only, no workflow. The addressing the
  product thesis requires, and nothing else: Dataset (stable id, name,
  tenant), DatasetVersion (immutable, content-addressed manifest of
  (example_id, digest) pairs with a parent version for lineage and the
  promotion's provenance — re-POSTing identical content IS the same
  version), Example (stable id across versions; input, optional
  expected output, split label, provenance back to the source span; bodies
  carry $payload references that count as live for the TTL sweep and
  reference-aware erasure, so a promoted copy is real for offloaded
  content), Experiment (stable id, one dataset version, config
  metadata), Run (the experiment→trace link, appended by the external
  harness), and Score — an annotation whose addressing was generalized
  to a typed subject (trace / span / session / experiment example), so a
  score addresses the (experiment, example, span) tuple with every
  existing annotation field preserved. All of it lives in evals.jsonl, a
  new manifested append-only recovery domain with the annotation log's
  torn-tail healing and pin-by-copy discipline. Deletion semantics were
  settled up front and are enforced by test: erasing source traces never
  corrupts a dataset version (the receipt's new eval-records domain
  REPORTS surviving copies and turns inconclusive — purging a curated copy
  is a deliberate second act); erasing a payload leaves dangling addresses
  in example bodies, reported retained-by-design without losing
  conclusiveness; a dataset-version tombstone is logical deletion with
  defined effects (410 with the tombstone, dependent experiments keep
  working and say why, new experiments refused); a tenant erasure takes the
  tenant's eval records inside the barrier, and ids are never reused past
  the rewrite (a counter record floors the allocators). Score distributions
  (/summary) and experiment-over-experiment diffs (/diff) come from
  ordinary reads with per-(example, name) last-write-wins dedup. The
  whole loop — promote failing traces, run externally, record runs and
  scores, read distributions and diffs — runs end to end in CI against the
  built binary, and survives kill -9.

• One recovery domain: generations and checkpoints. Query-visible state
  lived in several independent recovery domains — the write-ahead log and
  buffer, segments, annotations.jsonl, payloads/ — each with its own
  durability rule and its own idea of "now", and nothing named a state they
  all agreed on. That is why backup, export, retention and deletion were four
  mechanisms rather than one. A generation is that agreed state: a
  manifest naming every load-bearing file with its SHA-256 digest, plus the
  log position it folded through. CURRENT names the live one, and moving it
  — a staged rename made durable by a directory fsync — is the single commit
  point for a checkpoint, a restore, or a published deletion.

  • Backup runs against a live server: POST /v1/backups/{label}
    checkpoints, hard-links the manifested files into pins/{label}, and
    verifies every digest before reporting success. Hard links share inodes,
    so a pin costs almost no disk and holds its bytes even after compaction
    unlinks the originals — the copy proceeds at its own pace, and
    POST /v1/backups/{label}/release frees it. The pinned set carries
    spans, annotations and payload bytes together, which is what a span
    export never could.
  • Restore is install: traza-server --restore DIR (or Store::restore)
    verifies the backup before swapping anything, and commits at one
    CURRENT rename, so a failed or interrupted restore leaves the prior
    store rather than a blend.
  • GET /v1/verify re-digests the live generation and names each
    discrepancy, so recovery can distinguish damage it may ignore from damage
    that changes what the store contains by asking rather than inferring it
    from whether parsing happened to succeed.
  • POST /v1/checkpoint, plus a checkpoint every five minutes from the
    maintenance thread. Cheap by construction: segments are immutable, so a
    checkpoint carries their digests over from the previous manifest and
    hashes only what was written since.

• Log frames carry the generation they belong to. The framing gains an
  eight-byte magic and a (epoch, sequence) stamp — a 24-byte header, up from
  8. CURRENT and the log are separate filesystem objects, so no rename can
  make "the new generation is live" and "the frames folded into it are gone"
  one event; a crash between them meets a log still holding folded frames, and
  for a checkpoint that published a deletion, replaying them resurrects
  exactly what was deleted. Recovery now replays a frame only when its stamp
  is strictly after the live generation's folded_through, which demotes log
  reclamation from correctness to housekeeping. Sequences are monotonic for
  the life of the store — a rewrite keeps counting rather than restarting —
  because a stamp landing at or before a recorded fold point would be
  discarded by the next replay as though it had been folded.

  tests/generations.rs proves it against the crash state itself: the log's
  pre-checkpoint bytes are written back over a committed generation, and the
  test fails when the stamp rule is removed.

• Existing data directories are adopted at first open. Nothing moves —
  segments stay exactly where they are, because their paths are load-bearing —
  and generation one is published over the files already there, converting a
  pre-generation log to stamped framing on the way. One-way, resumable, and
  committed by the CURRENT rename, so a crash mid-adoption leaves a
  directory the next open finishes.

• Targeted deletion, with a receipt. POST /v1/erasures erases a
  subject — a trace, a span, a session (resolved across every recognized
  session key), or one offloaded payload — and the new
  traza-server verify --erasure (or GET /v1/erasures/{id}/verify) then
  proves it: every domain the subject's bytes could inhabit, checked by name,
  with the result of each. TTL removes by age; this removes because someone
  is entitled to have it gone, and the receipt is the difference.

  • The tombstone log (tombstones.jsonl) is a new manifested,
    append-only recovery domain beside the annotation log. The intent record
    is fsynced before anything is removed, so from that moment the subject
    is invisible to every read path — search, lookups, sessions, analytics,
    annotations, payload fetches, export, the live tail — even before the
    rewrites run, and a crash mid-purge leaves a pending erasure the next
    open masks and the maintenance tick finishes. Idempotent to resume: the
    purge re-verifies rather than re-damages. (The settle record's counts
    are therefore the settling pass's counts; the receipt, not the tallies,
    is the authority on absence.)
  • A pending erasure is an admission barrier, not just a veil. Covered
    spans are dropped at ingest — acknowledged, never stored, counted in
    traza_erasure_spans_suppressed_total — and the barrier is total: it
    runs BEFORE payload offloading, so a suppressed span leaves no orphan
    payload bytes behind (a review probe planted exactly that orphan — the
    file of a span that never entered the store, invisible to record and
    receipt alike); an oversized value whose content hash is itself under
    erasure offloads directly to its redacted marker instead of recreating
    the file being deleted; and annotations addressed to a covered subject
    are dropped at admission the same way. The barrier holds at the BEGIN
    transition, not just in the steady pending state: admissions hold an
    erasure gate in read mode from their mask load through their store
    mutation — the offload's file writes included — and begin/settle move
    the mask only under its write mode, so a batch or annotation is wholly
    before an erasure or wholly after it, never astride it. A one-shot mask
    check could not say that, and review was right to test the transition.
    The cut is exact: every span acknowledged before settled_unix_ns is
    erased or was never stored, under concurrent writers hammering the
    subject (a review probe demonstrated 92 pre-settle survivors before the
    barrier existed; tests/erasure.rs keeps the probes as regression
    tests, plus a transition stress test that erases in a loop under
    payload-writing and annotating threads and then audits the payload
    directory itself for orphans). The ingest
    response tells the truth about it — accepted counts what was stored,
    suppressed appears when nonzero, and the OTLP surfaces answer with
    partialSuccess.rejectedSpans (hand-encoded for the protobuf path,
    because an empty response is a claim of full success).
  • Nothing rewrites a manifested file after the checkpoint the settle
    cites. The confirm purge and annotation drops moved BEFORE the
    checkpoint; the settle append itself rides the append-only allowance
    every manifest grants the tombstone log. A review probe caught the old
    order making the settle's own generation fail verification
    (annotations.jsonl: digest mismatch) the moment a concurrent
    annotation landed; the ordering is now barrier → purge → confirm →
    checkpoint → settle → lift, and the suite asserts the cited generation
    verifies clean, including under concurrent writers.
  • The purge reaches every domain. Buffer and write-ahead log rewritten
    to the survivors (the TTL discipline: a deletion a restart undoes is not
    a deletion); segments rewritten in place, superseded versions of an
    erased key included — the purge tests every physical record, not the
    visible one; annotations addressed to erased spans dropped; payload
    files deleted reference-aware, because content addressing means one
    file can back spans outside the subject — those bytes are retained and
    the receipt names the reason. A payload subject additionally rewrites
    every referencing span to drop its inline preview: the preview is
    content too. Publication is a checkpoint — the deletion is durable when
    CURRENT moves, and tests/erasure.rs proves it by killing the server
    after its 200.
  • The receipt is a verification, not a claim. Its result is computed
    from the walk, never from the settle record. Matches are classified
    against the erase record's resolved keys under one rule for every
    domain: an erased key found live again is a re-delivery and fails the
    receipt; a fresh key under the same identifiers is new activity,
    reported without failing it — an erasure is a barrier, not a ban. Pins
    are checked and named: a backup pinned before the erasure still holds
    the bytes in its hard-link farm, and the receipt says which pin to
    release. What remains afterwards is stated rather than hidden: the
    tombstone record itself keeps the subject's identifiers and content
    hashes — never the erased text — as the record the receipt verifies
    against. Where a check is over-approximate by construction (the
    byte-level occurrence scans), its findings never blend into the
    verdict: the receipt carries a separate conclusive flag, and the
    subcommand exits 0 erased-and-conclusive, 3 erased-but-inconclusive,
    2 not erased.
  • Pins no longer share the append-only logs' inodes. A pin hard-links
    the immutable files and copies annotations.jsonl and
    tombstones.jsonl at their manifested length. Hard-linking them let
    every later append edit the "backup" in place — a pin taken before an
    erasure inherited the erasure's settle record through the shared inode,
    so restoring it produced a store that recorded a deletion it did not
    contain.
  • Erasure requires a new admin token scope. TRAZA_TOKENS grows
    admin: alongside rw:/ro:; POST /v1/erasures refuses plain rw
    with a 403. Every collector holds a write token, and a credential minted
    to produce telemetry must not be the credential that destroys it.
    Payload subjects are canonicalized to lowercase hex before anything is
    resolved or recorded — an uppercase hash previously matched no stored
    reference and produced a green receipt over untouched content. Redaction
    markers are no longer counted as live payload references anywhere
    (protection sets, receipts): a marker records that content is gone.
  • No MCP tool for any of it, deliberately. Deletion is an HTTP verb
    behind the admin scope; the agent-facing surface stays read-only, so
    stored adversarial text has no destructive tool to actuate.

• Derived LLM cost from a configured pricing table, --pricing FILE.
  OpenTelemetry defines no cost attribute, so a span carries one only if its
  pipeline metered it — and most do not, which left stores that knew the model
  and both token counts reporting $0.00 on every cost surface. Rates are USD
  per million tokens, keyed by exact model name or a prefix* pattern
  (longest match wins; an exact name beats every pattern; a bare "*" is a
  default). There is no built-in table and will not be one: prices move on the
  vendor's schedule, and self-hosted models have no public rate.

  • A metered llm.cost_usd always wins, and a span reporting only a
    total token count stays unpriced rather than being split by an assumed
    input/output ratio.
  • Estimates are reported as estimates, by count rather than by amount.
    Every cost-bearing row — /v1/stats/llm, /v1/sessions, and each bucket
    of /v1/stats/series — carries cost_derived_usd beside cost_usd plus
    cost_metered_calls, cost_derived_calls and cost_unpriced_calls. The
    dollars cannot carry provenance on their own: a zero-rate model is priced
    and adds nothing, a call nothing could price also adds nothing, and
    neither is distinguishable from a genuine measurement of zero.
    cost_derived_calls > 0 means the total is an estimate;
    cost_unpriced_calls > 0 means it is an undercount. The dashboard and MCP
    both mark an estimate ~, show — rather than 0.0000 when nothing
    could be priced, and state the breakdown — and analyze_cost no longer
    claims cost is exact when a rate table contributed to it.
  • Rollup sidecars record the fingerprint of the table they were folded
    under (format v4), so editing the rates invalidates exactly the cached
    counters that would now be wrong instead of reporting last month's prices
    from a sealed segment forever. An empty table fingerprints to zero, so a
    store that prices nothing binds its sidecars exactly as before. A
    malformed pricing file refuses startup rather than being ignored.

Changed

• The erasure receipt's failing verdict is not-erased, not incomplete.
  A receipt over a pinned backup reported result: "incomplete" beside
  conclusive: true — semantically right (the walk conclusively found the
  pin holding the bytes) but reading as a contradiction, because
  "incomplete" invites attaching to the verification when it describes the
  erasure. The value now names its subject: conclusively not-erased and
  erased-but-inconclusive both read as the claims they are, the wire value
  matches the prose the docs and the verify usage text already used
  (2 not erased), and the exit codes are unchanged.

• Ingest rejects an inadmissible tenant with 400, on both surfaces. A
  tenant is identity: lowercase [a-z0-9][a-z0-9._-], at most 64 bytes, or
  empty for the default. A misconfigured traza.tenant resource attribute
  fails the whole OTLP export loudly rather than being silently dropped —
  partialSuccess is for data the server chose to suppress, not for a defect
  the client must fix.

• Cursor tokens carry a version byte. The ordering key changed shape
  (tenant joined it), and a pre-tenancy token must parse as invalid, never
  as a plausible wrong position. Live cursors from before an upgrade get a
  400, which is what a stale cursor always deserved.

• The span identity key on the wire and on disk is $tenant, not
  tenant. A span's top-level namespace is open — unknown fields survive
  in the round trip — so a bare tenant is client data and must stay client
  data. The $ sigil (as in $payload) is the format discriminator: bytes
  written before tenancy cannot carry $tenant, so a store read after an
  upgrade keeps its bare tenant values as data rather than promoting one to
  an identity no query selects and no erasure names. This replaces an earlier
  decode-time normalization that tried to tell legacy data from identity by a
  value's shape — which a valid-looking legacy value ("acme") slipped
  through. The ?tenant= read filter and the erasure subject's tenant
  field are closed namespaces and keep the plain name — but they, along with
  an annotation's and a dataset's tenant, now also accept $tenant as an
  alias, so a client that learned the span's spelling cannot silently
  misroute a score, a dataset, or — worst of all — an erasure to the default
  tenant. The span alone rejects a bare tenant as identity, because the span
  alone has the open namespace that makes it data.

• Append-only files are digested and verified over their recorded prefix,
  exactly. digest_engine recorded an append-only log's length from one
  instant and its hash from another (sha256 over the live file), and
  verify_against hashed the whole file whenever the lengths happened to
  match — so an annotation or settle record appended mid-walk made a
  freshly published generation fail its own verification with a digest
  mismatch that was growth, not damage. Both sides now hash exactly the
  recorded length, which is what the checkpoint's concurrency story always
  claimed. Found by the erasure transition stress test; the race predates
  erasure and was reachable by any annotation landing inside a checkpoint's
  digest walk.

• wal_bytes counts replayable work, not file bytes. The log's constant
  preamble is excluded, so a reclaimed log measures zero. Both the
  flush_wal_bytes bound and the statistic mean the same thing — what a
  restart would redo — and a constant is neither replayed nor reclaimable.

• The checkpoint's commit point is proven by SIGKILL, not only
  deterministically. tests/durability.rs kills a real server mid-checkpoint
  at three named on-disk signals — the manifest directory appearing,
  .CURRENT.tmp existing between the staged write and its rename, and
  CURRENT itself changing — because those windows are single fsyncs wide
  and a stagger alone lands wherever the machine favours. Recovery must then
  answer with exactly one state: every acknowledged span present, the hot key
  at the newest acknowledged version, the live generation verifying clean
  over GET /v1/verify, a complete manifest behind whatever CURRENT names,
  and nothing staged surviving the sweep. Publishing CURRENT before the
  manifest is durable fails it.

  The staged-CURRENT signal exists because a slower CI runner reached that
  window when a fast machine never did — and a test that only covers a state
  when the machine happens to be slow is a test that passes for the wrong
  reason. Aiming at it beats depending on the weather.

  Recorded because the opposite would be a claim rather than evidence: the
  other ordering rule — reclaiming folded frames only after CURRENT is
  durable — was mutated too and did not fail. A checkpoint seals the
  buffer first, so the frames it reclaims are already in a durable segment,
  and recovery loads segments by walking the directory rather than by reading
  the manifest. That rule is defensive under today's engine and becomes
  load-bearing the moment segment loading is manifest-driven, which is what a
  replicated snapshot install wants. Invariant 12 now says so.

• Checkpointing is never a side effect of a primitive. It seals the write
  buffer, and expiry must not decide when to seal: expire_before deletes
  from every domain and stops, exactly as before. The deletion is durable when
  its domains are durable and published by the next checkpoint — one
  maintenance interval away, or immediately when a backup asks.

Fixed

• The bound-token payload probe now heals the sidecar it rebuilds. The
  reachability check behind scoped GET /v1/payloads/{ref}, MCP
  get_payload, and dataset-version promotion read rollup sidecars directly,
  so a segment missing one — a crash, a copied data directory, a pricing
  change invalidating every binding — was decoded again on every probe,
  forever. The probe now reads rollups through the same
  cache-then-sidecar-then-rebuild path every aggregation uses: the decode is
  paid once, the sidecar is written back, and the new
  traza_rollup_builds_total counter says whether heals are landing.

• A panicking handler permanently consumed a --max-connections slot.
  The live-connection count was decremented at the end of the handler
  closure — a line a panic never reaches — so each handler panic leaked one
  slot, and enough of them walked the server to a permanent 503 with
  nothing left in the logs but old panics. The slot is now claimed by an RAII
  guard whose Drop releases it on return, unwind, and spawn failure alike,
  and each unwind is counted in the new traza_http_handler_panics_total
  (any nonzero value is a bug report waiting to be filed).

• Two documentation claims that were not true of the code. The OTLP mapping
  table said "other resource and scope attributes" were merged beneath span
  attributes. Scope attributes are; resource attributes are discarded —
  only service.name and traza.tenant are read — so
  deployment.environment, service.version, host.name and k8s.* never
  reach the store and cannot be filtered on. The table now says so, names the
  workaround (copy them onto spans in an SDK processor or a Collector
  transform), and an OTLP conformance assertion pins the behaviour so the
  page cannot drift from it again. Separately, the MCP tool table billed
  record_annotation as "Score this trace"; it writes a trace/span annotation
  and cannot address an (experiment, example) pair, which is what a score is.
  The MCP guide now states that and that the eval entities are HTTP-only.

• A promotion could read across a tenant boundary. promote_failures_to_dataset
  pinned the tenant it wrote but not the tenants it read: an unbound credential
  resolves a session across every tenant, so naming one tenant's session copied
  its span attributes and provenance into a default-tenant dataset — and a
  promoted example deliberately outlives its source, so erasing that tenant
  afterwards left the copy standing, permanently outside the reach of the
  erasure that should own it. Every read the tool makes is now scoped to the
  tenant being written (Some("") being the default tenant named, not the
  absence of a scope). Diagnosis is unchanged: an operator may still read any
  session, and simply cannot copy one out of its tenant.

• Prompt-cache counters were recognized under one spelling. OpenLLMetry
  moved them to dotted segment names (gen_ai.usage.cache_read.input_tokens,
  gen_ai.usage.cache_creation.input_tokens) to match OTel GenAI, and only the
  older underscore forms were read — so on a cached Anthropic agent, the exact
  configuration this field exists for, the detector fell back to the uncached
  remainder and reported a growing context as ordinary iteration. Both
  spellings are accepted now, dotted first.

• Link attributes were outside every payload walker. A $payload
  reference copied into a span's links[].attributes — an ordinary thing for
  an SDK to do, since the wire contract stores what a client sends — was
  counted by no reference sweep, redacted by no erasure, and invisible to the
  verify predicate. So a blob a surviving span still referenced could be
  deleted, and verify --erasure could report erased and conclusive over
  content still on disk and still readable. All five walkers now share one
  iterator naming every attribute map a span carries; the duplicate collector
  that made the gap exist twice is gone. Links are also bounded at ingest for
  the first time, with its own test. The two erasure guards are
  mutation-proven: with the link arm removed from the iterator, one test finds
  the whole secret sitting inline in the link and the other finds the payload
  deleted while a live reference remained.

• top_failures and slowest_spans advertised a row limit whose maximum
  (100) and default (20) were both wrong — they apply 50 and 10 — so a caller
  obeying the schema was silently given a fraction of what it asked for.

• The claim in the data-model and LLM-semantics guides that a link's
  relation attribute "keeps link semantics queryable" was false — no query
  path reaches link attributes. The docs now say what is true: links are
  traversed inside a diagnosis, and are not filterable.

• A pending whole-tenant erasure withholds its payload bytes from a scoped
  fetch. A tenant erasure discovers its span-held references only as its
  purge walks them, so the mask's payload_files set fills after the mask
  already hides the tenant. A GET /v1/payloads under a bound credential now
  returns nothing the moment its tenant is masked — whether or not that exact
  reference has been enumerated — closing a window in which a planted crash
  state served the bytes an erasure was seconds from deleting. Freshly
  discovered references are also folded into the live mask as they are noted,
  so the operator path stops serving them mid-purge too.

• The MCP describe_store "nothing to search yet" note follows the
  caller's own usage, not the store total. For a tenant-bound caller the
  note was keyed on the store's total_records, so it appeared only on a
  globally empty store and vanished the instant any other tenant ingested a
  span — a co-tenant presence oracle across the isolation boundary the rest
  of the overview respects. It now reads the bound tenant's own row.

• An explicit default-tenant scope reaches its own sealed payload. The
  default (empty) tenant carries no attribute-index posting — that is what
  keeps single-tenant stores byte-identical — so a payload-reachability probe
  that trusted the posting found nothing for a Some("") scope once the span
  sealed. The probe now scans the records and lets the decoded tenant decide
  for the default tenant, exactly as select_probe already did.

• Resolving last-write-wins no longer re-reads the store to prove keys were
  never replaced. Every query-side supersede probe — the limited merge, the
  unlimited scan, and the fold behind the aggregation routes — now consults
  each newer segment's own key-hash set before paying an exact probe, the
  prefilter the analytics fold's exact path already ran; that fold now gates
  per segment too, so a key rewritten eleven segments later costs one probe
  rather than a walk across the ten between. On a store carrying superseded
  versions — a crash-recovered store before its first compaction is the
  canonical case — queries cost matches × segments × trace-width decodes,
  which is how a recovery query blew a 30-second deadline in CI. The new
  traza_supersede_probes_total counter is the observable: roughly one probe
  per superseded version actually held.

• A pre-$tenant rollup sidecar is rebuilt, never believed. Reserving
  $tenant changed what a bare tenant field decodes to, and a sidecar's
  key hashes are evidence only under the decoding they were computed with —
  trusting one across that boundary treated a stale membership miss as proof
  of absence and resurrected a superseded span. The rollup SCHEMA_VERSION
  is bumped, so the first read after upgrading rebuilds each segment's
  sidecar once; tests/fixtures/pr50-tenant-identity, sealed by the
  pre-reservation build, is the corpus that fails if a future decoding change
  forgets the bump.

• Native ingest accepts an event's timestamp under the OTLP name. A span's
  timestamps accepted start_time_unix_nano; its events accepted only
  timestamp_ns, so a client spelling both the way OTLP spells them had its
  entire batch rejected with a 400 naming a field it had supplied. Events
  now take time_unix_nano, timestamp_unix_nano, time_ns and time as
  aliases, and attributes defaults, so a named instant no longer needs an
  empty map to be legal. The failure mode this closes is quiet rather than
  loud: telemetry clients are conventionally fail-open, so the spans simply
  never arrived.

• The span search's token column reads the current OpenTelemetry names.
  It carried its own third copy of the semantic-convention precedence, which
  had drifted from src/semconv.rs: it resolved only the deprecated
  gen_ai.usage.{prompt,completion}_tokens and a llm.usage.prompt_tokens
  key Traza has never recognized. A span using the current input/output
  names — resolved correctly by the server and by the trace detail — showed a
  blank cell. It now uses the shared llmUsage helper, as everything else
  does.

• Two toolbar actions on the span search worked in name only. A local
  named window shadowed the global for the whole component, so
  window.prompt and window.location read undefined through the optional
  chains guarding them: saving a view never asked for a name and silently
  numbered every one view N, and "Copy as curl" emitted a hostless URL that
  curl refuses.