full-chart

Topic-to-archive agent pipeline. Takes a topic, finds authoritative sources, extracts time-series (or category-series) data, archives it with provenance, and enriches it with timeline annotations.

Implements the design in pipeline_design.md (Stages 1–5, the schemas, the extractor sub-pipeline pattern). LLM calls go through the llm package with the OpenRouter backend.

Install

pip install -r requirements.txt
# or
pip install -e .

Configure the OpenRouter key

Either:

llm keys set openrouter
# paste your sk-or-... key

or set the environment variable:

export OPENROUTER_KEY="sk-or-..."

The default model is openrouter/anthropic/claude-sonnet-4.5; override with --model openrouter/<provider>/<model> or PIPELINE_MODEL=....

Run

python -m pipeline "driver deaths per km in the US" -vv
# or after install:
full-chart "GDP per capita by country in 2024" --model openrouter/openai/gpt-4o

Outputs land in workdir/<slug>/:

workdir/driver_deaths_per_km_in_the_us/
  scope.json
  sources.json
  snapshots/src_*.{csv,json,html,pdf,xlsx}
  extracted/src_*.json     # one per source (or *.error.json on failure)
  archive.yaml             # the canonical archive
  state.json               # checkpoints for resumability

The pipeline is resumable: rerunning with the same topic skips stages that already produced their on-disk artifacts. Extraction is per-source, so a crash on one source doesn't lose progress on the others.

Stages

1. ScopeAgent — turns the topic into an axis type, unit, geographic scope and a list of target keywords.
2. SourceDiscoveryAgent — proposes 3–8 authoritative sources (primary agency first, then aggregators) with format hints.
3. ExtractorAgent (fan-out, one per source) — fetches the URL into snapshots/, dispatches to a format-specific sub-pipeline (CsvPipeline / JsonApiPipeline / HtmlTablePipeline / ExcelPipeline / PdfTextPipeline) which renders a chunk and asks the LLM to pull the target series.
4. ConsolidatorAgent — merges per-source series by x value, computes cross-source agreement, and asks the LLM only for the human metadata (display names, citations, license guesses). Numeric data is never touched by the LLM at this stage.
5. EnrichmentAgent — writes annotations to archive.yaml. Reruns are safe; it only adds, never deletes, and never touches data or sources.

All five axis types from the design (temporal, numeric_continuous, ordinal, nominal) flow through the same code path — x is treated as opaque.

Schemas

JSON Schemas live in schemas/:

• sources.schema.json
• extraction.schema.json
• archive.schema.json

Every stage validates its output against the corresponding schema before writing.

Extending sub-pipelines

pipeline/extractors/__init__.py:dispatch declares the escalation ladder. To add a new extractor (e.g. headless browser, vision PDF), subclass SubPipeline from pipeline/extractors/base.py, return an ExtractionResult, and append the class to the ladder for its format key.