Architecture

Architecture

A small, layered, quasi-stdlib pipeline. Raw JSONL logs are parsed into typed records; costs are computed at read time from those records; the same analytics layer feeds the CLI, the dashboard, and the CSV exports.

Data flow

flowchart TD
    A["~/.claude/projects/**/*.jsonl<br/>(CLAUDE_CONFIG_DIR honored — paths.py)"]
    A -->|"extract.collect() — streaming parse, per-file cache,<br/>global dedup on message.id+requestId,<br/>uuid→parentUuid attribution, filtering"| B
    B["schema.py shapes<br/>Session / Prompt / Token / Request rows (raw counts)"]
    B -->|"extract.run_extract() — atomic_write_csv"| C["6 normalized CSVs in ./output<br/>+ extract_meta.json (--since/--until window)"]
    B --> D["analytics.Dataset<br/>(sessions × prompts × tokens × requests × categories)<br/>+ CostEngine(provider): raw counts + pricing.yml → USD"]
    C --> D
    D --> E["cli.py<br/>(rich tables / json / csv)"]
    D --> F["dashboard/<br/>(pandas reshape, ECharts, Streamlit)"]
    D --> G["export --flat<br/>(one row / prompt)"]

Loading

The key decision: extract is an export, not a prerequisite. Every analysis command calls analytics.load_dataset(output_dir), which uses the output CSVs when they are fresh (CSV mtimes newer than the JSONL, current-schema headers) and otherwise parses the JSONL live in memory — fast thanks to the per-file parse cache. So the numbers are always current, and changing pricing.yml never requires a re-extract because costs come from raw counts at read time.

Modules

Module	Responsibility
schema.py	The data contract: token-type keys, every CSV's columns, the TypedDict shapes exchanged between parser and aggregation. Single source of truth.
extract.py	collect() parses JSONL → typed records; run_extract() adds atomic CSV writes + the extraction report. Owns the per-file parse cache.
pricing.py	Loads/validates data/pricing.yml; get_model_pricing() (exact→longest-prefix lookup, suffixes stripped), get_per_request(), load_plans(), load_copilot_plans(), is_long_context(). Grid sections: providers (per-token + optional per_request), plans, copilot_plans, updated_at.
analytics.py	Dataset, load_dataset/dataset_from_csvs, CostEngine, and every aggregation (summary, by_project, session_depth, context_growth/ttl_losses/compactions/session_overhead, model_category/recommendations/burn_rate, break_even, compare_providers, the composition views output_composition/context_cost/file_footprint/by_task/task_graph, the before/after impact_report/split_on_pivot, flat_export…). No streamlit/pandas.
categorize.py	Weighted FR+EN regex classifier (heuristic-v2, thirteen categories) + observed complexity 1–5; the opt-in semantic classifier (SemanticClassifier, shared data/semantic_anchors.yml); LLM clients (Anthropic / OpenRouter / Ollama / Azure). Writes categories.csv.
compose.py	Output composition (Axe C): language by extension, code/test kind, exact LCS line diff, prose-vs-code split. Metrics only — never the diff content.
context.py	Context composition (Axe D): the 4 source taxonomy + the attribute_context_cost walk that splits loading (cache_write) from rent (cache_read), reconciled to the bill.
tasks.py	Task attribution (Axe B2): assembles tasks from the TodoWrite spine with an inference fallback (gaps + embeddings + category structure); maps each prompt to one task.
tokenizer.py	Local token counting (tiktoken at the core, offline heuristic fallback) — powers the prose/code and per-source size splits.
embeddings.py	The single door to the vector model (Axe B1): Embedder Protocol, StaticEmbedder (model2vec, torch-free, offline), HashingEmbedder (test double), .npz cache.
taxonomy_audit.py	categorize --audit-categories: clusters the whole corpus (HDBSCAN) into an alignment matrix vs the thirteen categories. Diagnostic only — never writes categories.csv.
snapshot.py	Reads the OAuth quota endpoint; appends to quota_log.csv.
paths.py	The two machine-dependent roots: the Claude dir (CLAUDE_CONFIG_DIR honored) and the per-user cache dir (%LOCALAPPDATA% / $XDG_CACHE_HOME).
storage.py	atomic_write_csv/atomic_write_json (tmp + os.replace), CSV formula-injection escaping, append-with-header-if-empty.
config.py	Pure read of config.yml (+ config init).
render.py	Turns an analytics TableResult into a rich table / CSV / JSON. The only place display formatting ($, %, separators) lives.
cli.py / __main__.py	Argument parsing, dispatch, exit codes.
dashboard/	Streamlit app: data.py (Dataset → pandas frames), filters.py (cross-filters), echarts.py (Apache ECharts render + click/brush), theme.py (colors + typography), impact.py (the Compare tab's before/after panel, built from analytics.impact_report), one script per page under pages/ (incl. 8_composition.py, the two Explorers). Optional extra.

The data model (CSV contract)

The canonical output is the normalized CSVs below (+ categories.csv from categorize, quota_log.csv from snapshot, and the extract_meta.json marker), keyed by prompt_id / session_id. The authoritative column layout is prompt_analytics/schema.py — every reader and writer imports from there, so adding a column is a one-line change. The composition tables (Axes C/D/B2) are additive: prompts.csv/tokens.csv stay byte-identical, and all of them keep metrics only (no source code, no absolute paths).

File	Key / grain
sessions.csv	one session
prompts.csv	one real human prompt (pseudo-prompts excluded)
prompts_text.csv	full prompt text (omitted with --no-text)
tokens.csv	prompt_id × model × token_type × is_sidechain — raw counts only, no costs
requests.csv	prompt_id × request_index — one deduplicated API request
token_types.csv	reference: machine key, label, description
categories.csv	category, complexity, classifier model (owned by categorize)
output_files.csv	output (C): prompt_id × path — language, code/test kind, lines +/−, edit count
output_tokens.csv	output (C): prompt_id — prose vs code token split (sums to the prompt's output)
context_sources.csv	context (D): session_id × source × language [× path] — context size in tokens, item count
context_cost.csv	context (D): session_id × source × language × model [× path] — rent (cache_read) and load (cache_write 5m/1h) tokens
tasks.csv	task (B2): one task — task_id, session, name, origin (todo/inferred), span
task_prompts.csv	task (B2): task_id × prompt_id edges (cost is derived at read time, never stored)
quota_log.csv	append-only quota time series

Key invariants:

• tokens.csv keeps session_id/model denormalized so pseudo-prompt usage (<session>:_continuation, no prompts.csv row) stays attributable/priceable.
• is_sidechain (0/1) splits subagent usage as a dimension — summing over it reproduces per-prompt totals (the V7 invariant: any schema change keeps per-prompt sums identical and re-reconciles with ccusage).
• requests.csv sums equal tokens.csv per prompt, exactly (tested).
• token_type is a machine key; human labels live in token_types.csv.

JSONL format (undocumented, evolving)

Parsing is pinned against fixtures per Claude Code version (tests/fixtures/claude-code-<version>/; capture with scripts/capture_fixture.py). Highlights:

• Progressive snapshots: one assistant message = several JSONL lines, each a growing snapshot. Rule: largest usage snapshot wins, ties → first line.
• Dedup key: message.id + requestId, global across all files (fixes resumed / --resume double-counting).
• Attribution: assistant events walk uuid/parentUuid back to a user prompt; unattributable tails → <session>:_continuation pseudo-prompt.
• Sub-agents (subagents/*.jsonl): cost rolled into the parent prompt.
• Filtered: isMeta, <command-*>, interruptions, post-compaction notice → out of prompts.csv, but their tokens still counted against the session.

See Accuracy and Pricing for the reconciliation and pricing, and Codebase Guide for how to navigate and change the code.