sonar

Observability for agent runs — an OpenTelemetry-style span tracer, token/cost/latency meters, a trace store, and text + HTML timelines, in ~2k readable lines of Python.

sonar is the layer you bolt onto an agent to see what happened and what it cost. It is not an APM vendor and not a framework: it is a tight, legible implementation of the observability plane every agent run needs — nested spans, deterministic timing, token/cost accounting, latency percentiles, a round-trippable trace store, and waterfalls you can read in a terminal or a browser.

Crucially, sonar consumes traces; it never calls a model. It reads what an agent already did — including the outputs of its sibling repos — and turns it into something you can inspect and cost.

---

Why this exists / what it demonstrates

The agent ecosystem is loud about running models and quiet about seeing the runs. This repo makes the seeing the point. Reading it should teach you

• the span model of an agent run — an AGENT root, LLM spans per turn, TOOL spans per call, nested by a context stack exactly the way the code nests;
• why timing should be injectable — the tracer times against a Clock, so a ManualClock makes every span duration an exact, asserted number. The whole test suite runs with zero real time and zero flakiness;
• how to cost a run honestly — each LLM span is priced against its own model attribute, so a mixed-model run is summed correctly rather than blended, and percentiles use a documented interpolation, not a hand-wave;
• how observability snaps onto the rest of a stack — adapters ingest a cogs record/replay cassette and a gauntlet eval result and reconstruct a span tree from each, so one renderer/meter serves all three repos.

sonar is the third repo in a trio: cogs is the agent runtime, gauntlet evaluates it, and sonar observes both.

---

Architecture

   cogs cassette ─┐                         ┌── to_text_timeline ── terminal
                  │   ┌───────────────┐     │
 gauntlet result ─┼──▶│    ingest     │──┐  ├── to_html ─────────── waterfall.html
                  │   └───────────────┘  │  │
   live agent  ───┘                      ▼  │
        │  with tracer.start_span(...)  ┌───────┐   ┌──────────┐
        └────────────────────────────▶ │ Tracer │─▶│  Trace   │──┼── to_markdown ── report.md
                     reads time from    └───┬────┘   │ (spans)  │  │
                            │               │ emits  └────┬─────┘  │
                     ┌──────────────┐       ▼             │        │
                     │    Clock     │  ┌───────────────┐  ▼        │
                     │ System|Manual│  │ JsonlTraceStore│ ┌──────┐ │
                     └──────────────┘  │  (JSONL round- │ │ meter│─┘
                                       │   trippable)   │ │ p50/ │
                                       └───────────────┘ │ p95, │
                                                         │ cost │
                                                         └──────┘

The Clock seam is the whole trick: swap SystemClock for ManualClock and the identical tracer produces byte-for-byte deterministic timings, which is what makes the offline tests and the timing-free ingest adapters possible.

---

Module map

Module	Responsibility
sonar/types.py	Frozen value types: Span, SpanKind, SpanStatus, SpanEvent, Trace (tree view over a flat span list), Usage, Cost. The lingua franca.
sonar/clock.py	The Clock protocol + SystemClock (monotonic) and ManualClock (deterministic virtual time). The seam that makes timing reproducible.
sonar/tracer.py	Tracer — start_span() context managers that nest via a stack, record start/end from the Clock, capture usage/events/errors, and emit finished spans to an exporter. Produces a Trace.
sonar/costs.py	Price table + usd_cost with anthropic.-prefix-tolerant lookup — deliberately identical to gauntlet.costs so ingested runs price consistently.
sonar/meter.py	Roll a Trace (or many) into a MeterSummary: counts by kind, error rate, tokens & USD, latency p50/p95 (documented interpolation), tool histogram.
sonar/store.py	TraceStore protocol + JsonlTraceStore — append spans as JSONL, reload into Traces, query by id. Exact round-trip, stdlib json only.
sonar/ingest.py	from_cogs_cassette and from_gauntlet_result — reconstruct span trees from the sibling repos' outputs, with synthetic deterministic timings.
sonar/render.py	to_text_timeline (ASCII waterfall), to_html (self-contained, no external assets), to_markdown (a gauntlet-voiced meter report).
sonar/cli.py	python -m sonar: show, html, report, ingest. All offline.

---

Quickstart

Everything runs offline — no credentials, no network. Uses uv; Python 3.11+.

uv venv
uv pip install -e '.[dev]'

1. Render a checked-in trace timeline

sonar ships a sample trace and can also reconstruct one from a cogs cassette. To render the checked-in sample:

uv run python -m sonar show examples/cassettes/sample_trace.jsonl

trace demo-session  (5.000s, 5 spans)
  agent:agent                        + 0.000s   5.000s |==============================|
    llm:llm-turn-0                   + 1.000s   2.000s |      ============            |  [420in/38out]
      tool:list_dir                  + 2.000s   1.000s |            ======            |
    llm:llm-turn-1                   + 3.000s   1.000s |                  ======      |  [640in/42out]
    llm:llm-turn-2                   + 4.000s   1.000s |                        ======|  [2100in/60out]

Convert a real cogs cassette into a sonar trace, then show or open it:

uv run python -m sonar ingest --cogs ../cogs/examples/cassettes/coding_session.jsonl -o run.jsonl
uv run python -m sonar show run.jsonl
uv run python -m sonar html run.jsonl -o waterfall.html   # self-contained, opens offline

2. Aggregate a directory of traces into a meter report

uv run python -m sonar report examples/cassettes/   # Markdown: counts, cost, p50/p95, tool histogram

# sonar report

## Overall
- Traces: 1
- Spans: 5
- Error rate: 0.0% (0 errored)
- Tokens: 3160 in / 140 out
- Est. cost: $0.0193
- Latency: p50 1.000s / p95 4.400s / max 5.000s (mean 2.000s over 5 spans)

3. Trace a live run in code

from sonar import Tracer, SystemClock, SpanKind, Usage, to_text_timeline

tracer = Tracer(clock=SystemClock())
with tracer.start_span("agent", SpanKind.AGENT):
    with tracer.start_span("llm", SpanKind.LLM) as s:
        s.set_attribute("model", "claude-opus-4-8")
        s.record_usage(Usage(input_tokens=420, output_tokens=38))
    with tracer.start_span("read_file", SpanKind.TOOL):
        ...  # run the tool

print(to_text_timeline(tracer.build_trace()))

See examples/trace_demo.py (fully offline, writes an HTML waterfall) and examples/ingest_cogs.py (ingests the sibling cassette).

---

Development

uv run ruff check .
uv run pytest

Both run fully offline. The suite times spans against a ManualClock, so durations are asserted to the microsecond with no sleep and no flakiness.

---

Intentional non-goals

This is a tasteful mini-implementation, scoped on purpose. What's left out is left out deliberately:

• Synchronous, single-threaded tracer. The current-span stack is a plain list, not thread-local or task-local. Instrumenting concurrent asyncio tasks or threads on one tracer would interleave spans incorrectly. A production tracer keys the stack by task/thread (or uses contextvars); that is a known extension point, kept out to stay readable — the sibling repos are synchronous too.
• No live export protocol. Spans are emitted to an in-process exporter and a JSONL store, not pushed over OTLP/gRPC to a collector. The OpenTelemetry data model is the influence; the wire protocol and a collector are not the point here.
• Estimated costs, not billing truth. The price table is Anthropic list prices; Bedrock and other resellers differ. sonar reports an estimate and says so — it is not a billing source of truth.
• Synthetic timings on ingest. A cogs cassette and a gauntlet result record what happened but not real per-step wall-clock timing, so the ingest adapters stamp deterministic virtual time (one second per step). Ordering and structure are exact; absolute durations are synthetic and labeled as such.
• No sampling, no retention policy, no UI server. Every span is kept; there is no head/tail sampling, no TTL, and no web app. These are straightforward to layer on the same Trace and TraceStore seams but would add surface area without changing what the repo demonstrates.

Each of these is a place a production system does more — and where this repo deliberately stops, so the core stays legible.

---

The platform

sonar is one repo in a five-part agent platform. Each owns a single concern, stands alone, and shares the same spine: a normalized, Bedrock-default provider seam and deterministic, fully-offline tests.

Repo	Concern
cogs	the agent runtime — the loop, tool protocol, provider seam, record/replay
bulkhead	reliable serving — a gateway (retries, circuit breaking, rate limits, caching, failover, budgets) in front of any provider
loom	context engineering — retrieve, compact, and assemble what goes in the window
sonar	observability — reconstruct a run as a cost/latency timeline ← this repo
gauntlet	evaluation — hermetic tool-use tasks scored with pass@k + confidence intervals

How work flows through them:

loom ──assemble context──▶ cogs ──model calls──▶ bulkhead ──▶ provider
                            │
              run cassette ─┴──▶ sonar (timeline, cost)
              eval result ─────▶ gauntlet (pass@k)

sonar is where the platform becomes visible: its ingest adapters read a cogs cassette and a gauntlet result directly, with no changes to those repos.

Run it combined

Check out the repos side by side (cogs, gauntlet, sonar under one folder), then from the sonar repo — everything below is offline, no credentials:

See a cogs run as a timeline. cogs records every agent run to a JSONL cassette; sonar turns one into a cost/latency waterfall:

uv run python -m sonar ingest --cogs ../cogs/examples/cassettes/coding_session.jsonl -o run.jsonl
uv run python -m sonar show run.jsonl                  # ASCII waterfall, per-turn tokens
uv run python -m sonar html run.jsonl -o waterfall.html   # self-contained, opens offline

Roll up a gauntlet eval. Produce a scripted (offline) eval run, then meter it:

( cd ../gauntlet && uv run gauntlet run --scripted --json-out /tmp/gauntlet.json )
uv run python -m sonar ingest --gauntlet /tmp/gauntlet.json -o gt.jsonl
uv run python -m sonar report gt.jsonl                 # Markdown: counts, cost, p50/p95

Note: gauntlet's --json-out is a headline dump, so the gauntlet → sonar rollup shows per-trace structure but not per-turn token detail (tokens read as 0). The cogs → sonar path carries full per-turn usage. bulkhead and loom share cogs's provider/message seam and drop into the loop directly; wiring them into a single end-to-end script is a few lines of glue, not a new interface.

---

License

MIT © 2026 Deepak