Codebase Guide

Codebase Guide

A practical map for reading and changing the code. If you only read one wiki page before touching the source, read this one — then Architecture for the why.

The shape to keep in your head: raw JSONL → typed records (schema.py) → analytics (pure Python) → {CLI, dashboard, CSV}. Costs are computed at read time; the analytics layer never imports Streamlit or pandas.

Module map

You want to change…	Look here
What a column is called / adding a column / the token-type keys	schema.py — the single source of truth, imported by every reader and writer.
How JSONL is parsed, deduped, attributed; the CSV writer	extract.py (collect() parses; run_extract() writes + reports). Owns the per-file parse cache.
A number in any report / a new analysis	analytics.py — Dataset, load_dataset, CostEngine, and every aggregation. No UI deps.
How a model is priced / adding a provider	pricing.py + data/pricing.yml.
How prompts get their category / complexity	categorize.py (regex classifier + opt-in LLM clients).
The quota endpoint / quota_log.csv	snapshot.py.
Where the Claude dir / cache dir live (machine-dependent roots)	paths.py.
Atomic writes, formula-injection escaping	storage.py.
How a TableResult becomes a rich table / CSV / JSON	render.py — the only place display formatting ($, %, separators) lives.
A command's args / dispatch / exit code	cli.py + __main__.py.
Anything in the Streamlit UI	dashboard/ — data.py (Dataset → pandas), filters.py (cross-filters), echarts.py (render + click/brush), theme.py (colors), pages/*.

The data model

Read schema.py first — it defines the token-type keys, every CSV's columns, and the TypedDict shapes exchanged between parser and aggregation. The canonical output is six normalized CSVs keyed by prompt_id / session_id:

File	Grain
sessions.csv	one session
prompts.csv	one real human prompt (pseudo-prompts excluded)
prompts_text.csv	full prompt text (omitted with --no-text)
tokens.csv	prompt_id × model × token_type × is_sidechain — raw counts only, no costs
requests.csv	prompt_id × request_index — one deduplicated API request
token_types.csv	reference: machine key, label, description
categories.csv	category, complexity, classifier model (owned by categorize)
quota_log.csv	append-only quota time series

Because schema.py is the single source of truth, adding a column is a one-line change there that every reader/writer picks up. The hard rule when you touch the schema: keep the V7 invariant — summing tokens.csv over is_sidechain must still reproduce per-prompt totals, and the numbers must still reconcile with ccusage (see Accuracy and Pricing).

Run, test, extend

Dev setup is uv-based. The four checks that gate CI (the "quartet"):

uv sync --all-extras
ruff check .
ruff format --check .
mypy prompt_analytics tests
pytest                       # coverage gate 85%

Run the tool against the demo data without your own logs:

prompt-analytics summary --from-csv demo_data
CCA_DEMO=1 prompt-analytics dashboard     # the dashboard on demo data

Adding a new analysis command

1. Write the aggregation in analytics.py, returning a TableResult (so it gets table / CSV / JSON for free via render.py). Keep it pure — no UI deps.
2. Wire the subcommand in cli.py (args + dispatch).
3. Optionally surface it in the dashboard under dashboard/pages/ using data.py to reshape the Dataset into pandas.
4. Add a test (the suite is the safety net for a refactor) and, if it shows a new number, keep the README/wiki example honest.

Adding a pricing provider

Add a rate card under providers: in data/pricing.yml (per-token prices, optional per_request). Lookup is exact-then-longest-prefix, so you don't need an entry for every point release. See Accuracy and Pricing for the lookup rules and the weekly drift job that keeps the bundled grid honest.

Supporting a new Claude Code version

Capture a fixture with scripts/capture_fixture.py into tests/fixtures/claude-code-<version>/. The parser is pinned against these, so a format change shows up as a failing test rather than silent drift.

Mental model for newcomers

• extract is an export, not a prerequisite. Analysis commands parse the JSONL live in memory when there is no fresh extract — so numbers are always current and a pricing change never needs a re-extract.
• One source of truth per concern: schema in schema.py, formatting in render.py, prices in pricing.yml, colors in dashboard/theme.py. When in doubt, change it in the one place.
• The analytics layer is UI-agnostic. If you find yourself importing Streamlit or pandas into analytics.py, the logic belongs in dashboard/ instead.

See Architecture for the data-flow diagram and the design decisions behind this layout.