Architecture

Architecture

A small, layered, quasi-stdlib pipeline. Raw JSONL logs are parsed into typed records; costs are computed at read time from those records; the same analytics layer feeds the CLI, the dashboard, and the CSV exports.

Data flow

flowchart TD
    A["~/.claude/projects/**/*.jsonl<br/>(CLAUDE_CONFIG_DIR honored — paths.py)"]
    A -->|"extract.collect() — streaming parse, per-file cache,<br/>global dedup on message.id+requestId,<br/>uuid→parentUuid attribution, filtering"| B
    B["schema.py shapes<br/>Session / Prompt / Token / Request rows (raw counts)"]
    B -->|"extract.run_extract() — atomic_write_csv"| C["6 normalized CSVs in ./output<br/>+ extract_meta.json (--since/--until window)"]
    B --> D["analytics.Dataset<br/>(sessions × prompts × tokens × requests × categories)<br/>+ CostEngine(provider): raw counts + pricing.yml → USD"]
    C --> D
    D --> E["cli.py<br/>(rich tables / json / csv)"]
    D --> F["dashboard/<br/>(pandas reshape, ECharts, Streamlit)"]
    D --> G["export --flat<br/>(one row / prompt)"]

Loading

The key decision: extract is an export, not a prerequisite. Every analysis command calls analytics.load_dataset(output_dir), which uses the output CSVs when they are fresh (CSV mtimes newer than the JSONL, current-schema headers) and otherwise parses the JSONL live in memory — fast thanks to the per-file parse cache. So the numbers are always current, and changing pricing.yml never requires a re-extract because costs come from raw counts at read time.

Modules

Module	Responsibility
schema.py	The data contract: token-type keys, every CSV's columns, the TypedDict shapes exchanged between parser and aggregation. Single source of truth.
extract.py	collect() parses JSONL → typed records; run_extract() adds atomic CSV writes + the extraction report. Owns the per-file parse cache.
pricing.py	Loads/validates data/pricing.yml; get_model_pricing() (exact→longest-prefix lookup, suffixes stripped), get_per_request(), load_plans(), is_long_context(). Grid sections: providers (per-token + optional per_request), plans, updated_at.
analytics.py	Dataset, load_dataset/dataset_from_csvs, CostEngine, and every aggregation (summary, by_project, session_depth, context_growth/ttl_losses/compactions/session_overhead, model_category/recommendations/burn_rate, break_even, compare_providers, flat_export…). No streamlit/pandas.
categorize.py	Weighted FR+EN regex classifier (heuristic-v2, 11 categories) + observed complexity 1–5; opt-in LLM clients (Anthropic / OpenRouter / Ollama). Writes categories.csv.
snapshot.py	Reads the OAuth quota endpoint; appends to quota_log.csv.
paths.py	The two machine-dependent roots: the Claude dir (CLAUDE_CONFIG_DIR honored) and the per-user cache dir (%LOCALAPPDATA% / $XDG_CACHE_HOME).
storage.py	atomic_write_csv/atomic_write_json (tmp + os.replace), CSV formula-injection escaping, append-with-header-if-empty.
config.py	Pure read of config.yml (+ config init).
render.py	Turns an analytics TableResult into a rich table / CSV / JSON. The only place display formatting ($, %, separators) lives.
cli.py / __main__.py	Argument parsing, dispatch, exit codes.
dashboard/	Streamlit app: data.py (Dataset → pandas frames), filters.py (cross-filters), echarts.py (Apache ECharts render + click/brush), theme.py (colors + typography), one script per page under pages/. Optional extra.

The data model (CSV contract)

The canonical output is six normalized CSVs (+ quota_log.csv and the extract_meta.json marker), keyed by prompt_id / session_id. The authoritative column layout is prompt_analytics/schema.py — every reader and writer imports from there, so adding a column is a one-line change.

File	Key / grain
sessions.csv	one session
prompts.csv	one real human prompt (pseudo-prompts excluded)
prompts_text.csv	full prompt text (omitted with --no-text)
tokens.csv	prompt_id × model × token_type × is_sidechain — raw counts only, no costs
requests.csv	prompt_id × request_index — one deduplicated API request
token_types.csv	reference: machine key, label, description
categories.csv	category, complexity, classifier model (owned by categorize)
quota_log.csv	append-only quota time series

Key invariants:

• tokens.csv keeps session_id/model denormalized so pseudo-prompt usage (<session>:_continuation, no prompts.csv row) stays attributable/priceable.
• is_sidechain (0/1) splits subagent usage as a dimension — summing over it reproduces per-prompt totals (the V7 invariant: any schema change keeps per-prompt sums identical and re-reconciles with ccusage).
• requests.csv sums equal tokens.csv per prompt, exactly (tested).
• token_type is a machine key; human labels live in token_types.csv.

JSONL format (undocumented, evolving)

Parsing is pinned against fixtures per Claude Code version (tests/fixtures/claude-code-<version>/; capture with scripts/capture_fixture.py). Highlights:

• Progressive snapshots: one assistant message = several JSONL lines, each a growing snapshot. Rule: largest usage snapshot wins, ties → first line.
• Dedup key: message.id + requestId, global across all files (fixes resumed / --resume double-counting).
• Attribution: assistant events walk uuid/parentUuid back to a user prompt; unattributable tails → <session>:_continuation pseudo-prompt.
• Sub-agents (subagents/*.jsonl): cost rolled into the parent prompt.
• Filtered: isMeta, <command-*>, interruptions, post-compaction notice → out of prompts.csv, but their tokens still counted against the session.

See Accuracy and Pricing for the reconciliation and pricing, and Codebase Guide for how to navigate and change the code.