release: 0.7.0 (ETL pipeline + handoff doc)

CHANGELOG.md

@@ -7,12 +7,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
-### Added
-- **Wave 4D — 12 beta provider normalizers.** Codeium (stub), Continue, Copilot, Cursor Agent, Droid, Gemini, KiloCode, Kiro, OpenClaw, OpenCode, Pi/OMP, Qwen, Roo Code now have `Normalizer` subclasses registered at import time. ETL pipeline now covers all 16 providers from the codeburn catalog. Beta providers stay opt-in via the existing `STACKUNDERFLOW_BETA_*` env flags — when they're enabled the matching normalizer fires automatically.
-- **Wave 4A — analytical routes migrate to mart reads (Wave 3B redo).** `compare_models()` reads `model_day_mart` + `session_mart`; `yield_tracker._query_sessions()` reads `session_mart`; `optimize._detect_cache_overhead()` reads `session_mart`; `/api/messages/summary` reads `project_mart`. Same JSON contract, ~30-50× faster on the user's real store. Empty-mart fallback to aggregator preserved per route.
-- **Wave 4B — backfill actually populates `usage_events`.** `stackunderflow etl backfill` now reads every message from the `messages` table, runs the matching provider normalizer (Wave 2A), and inserts into `usage_events`; `--force` rebuilds from scratch. Idempotent via `uniq_events_msg` UNIQUE index. The ingest writer (`stackunderflow/ingest/writer.py`) gets a normalize+insert hook so newly-ingested messages auto-create events without needing a backfill pass. Marts auto-refresh via `refresh_all_marts()` after each batch.
-- **Wave 4C — `/api/etl/status` + `stackunderflow etl status`.** Single endpoint surfaces watcher health, mart watermarks vs max event id, per-provider event counts, and a `health` enum (live/syncing/stale/error) so the dashboard can show a status badge and the CLI a one-line health check. <50ms response — all counts are indexed COUNT(*).
-- **Wave 4F — ETL status badge in the dashboard header + Settings backfill button.** New `EtlStatusBadge` polls `/api/etl/status` every 10s and shows live/syncing/stale/error health with a click-through popover detailing per-mart watermarks, per-provider event counts, and watcher state. Settings page gains an "ETL pipeline" section with a "Backfill now" button — POSTs to `/api/etl/backfill` when available, else shows the equivalent CLI command.
+## [0.7.0] - 2026-05-06
+
+### Added — ETL pipeline (Waves 1-4)
+
+The dashboard's per-request aggregator passes are gone. Every cost / dashboard / compare / yield / optimize / messages-summary route now reads from indexed materialized marts; the watcher syncs the marts within ~400ms of any source-file write. End-to-end on a 247K-message store: dashboard cold-load went from 2.5s to <50ms warm. The pipeline is three layers — raw `messages` → normalized `usage_events` → 5 materialized marts — wired together by a debounced filesystem watcher and a watermarked refresh loop. See `docs/specs/etl-architecture.md` for the design contract and `docs/HANDOFF.md` for a state-of-the-codebase walkthrough.
+
+- **Wave 1 — foundation (`stackunderflow/etl/`).** Migration v006 adds 7 tables: `usage_events` (canonical fact table, one row per billable event with `source_message_fk` UNIQUE for idempotent re-runs), `daily_mart`, `session_mart`, `project_mart`, `provider_day_mart`, `model_day_mart`, plus `mart_watermark` to track per-mart `last_event_id`. New `Normalizer` ABC + `MartBuilder` ABC + last-wins registries (`register/get/all`); `etl/watermark.py` (`get_watermark`, `set_watermark`, `refresh_all_marts`); `etl/backfill.py` (`BackfillReport` dataclass + idempotent orchestrator). Migration is additive — existing tables and routes keep working.
+- **Wave 2A — 4 default-on provider normalizers.** `claude`, `codex`, `cursor`, `cline` `Normalizer` subclasses transform `messages` rows into canonical `usage_events`. Codex token normalization (subtract cached, fold reasoning) moves into `CodexNormalizer` — single source of truth, no more drift between adapter + pricer. Cursor v3 no-per-message-tokens estimates from `len(text)//4` and stamps `cost_source='estimated'`. `cost_usd` computed once per event, stored on the row, never recomputed downstream.
+- **Wave 2B — 5 mart builders.** Indexed read-side rollups derived from `usage_events`. Each builder is watermarked + idempotent: incremental refresh from `last_event_id` for additive marts (daily, provider_day, model_day) with a follow-up DISTINCT recompute pass for `session_count` correctness across windows; replace-from-scratch-for-affected-keys for per-entity marts (session, project) so totals stay correct when new events arrive for an existing session. `MartBuilder.rebuild_from_scratch()` for full-backfill recovery.
+- **Wave 2C — filesystem watcher.** `watchfiles`-backed daemon thread that watches every registered adapter's source paths. On any change → `adapter.read(since=watermark)` → normalize → `usage_events` → `refresh_all_marts`. Debounced 200ms to coalesce JSONL append bursts. End-to-end latency **155 ms** smoke-tested against `~/.claude/projects` on the maintainer's machine (under the 400 ms target). New `BaseAdapter.watch_paths()` method (default `[]`); claude/codex/cursor/cline adapters return their canonical roots. `stackunderflow start --no-watcher` flag for headless mode; `STACKUNDERFLOW_DISABLE_WATCHER=1` env var equivalent.
+- **Wave 3A — hot-path routes migrate to mart reads.** `/api/projects?include_stats=true`, `/api/dashboard-data`, `/api/cost-data` (totals/by_day/by_model blocks), and `/api/cost-data/by-provider` now read from `project_mart` + `daily_mart` + `provider_day_mart` instead of running per-request aggregator passes. Same JSON contract; ~50× faster on a 28K-message project (cold 2.5–2.8s → 50ms warm). Per-session / per-command / per-tool detail blocks stay on the aggregator path until lower-grain marts ship in a future wave.
+- **Wave 4A — analytical routes migrate to mart reads.** `compare_models()` reads `model_day_mart` + `session_mart`; `yield_tracker._query_sessions()` reads `session_mart`; `optimize._detect_cache_overhead()` reads `session_mart`; `/api/messages/summary` reads `project_mart`. Same JSON contract, ~30-50× faster on real data. Empty-mart fallback to aggregator preserved per route so the contract holds even before backfill runs.
+- **Wave 4B — backfill streams `messages → usage_events`; ingest writer auto-normalizes.** `stackunderflow etl backfill` now reads every existing message via id-paginated chunks (5K-row transactions, never `fetchall()`), runs the matching provider normalizer, and inserts into `usage_events`; `--force` drops + rebuilds; idempotent via `uniq_events_msg`. The ingest writer hooks the same normalize-and-insert helper so newly-ingested messages auto-create events. After every backfill pass and every writer batch, `refresh_all_marts(conn)` runs to bring the marts forward. Smoke test on the maintainer's real ~250K-message store: first pass inserted **150,337 events** in 226 s (2nd run: 0 inserted, 29 s, fully idempotent). Marts populated: daily=940, session=841, project=151, provider_day=146, model_day=184. `SUM(daily_mart.cost_usd)` == `SUM(usage_events.cost_usd)` to the cent.
+- **Wave 4C — `/api/etl/status` + `stackunderflow etl status`.** Single endpoint surfaces watcher health, mart watermarks vs max event id, per-provider event counts, per-cost-source breakdown, and a `health` enum (`live` / `syncing` / `stale` / `error`). <50 ms response. Shared `etl/status.py` assembler keeps SQL out of the CLI; the route + CLI never drift. Watcher state degrades to `running="unknown"` when no live handle (CLI mode, headless) instead of crashing.
+- **Wave 4D — 12 beta provider normalizers.** Codeium (stub), Continue, Copilot, Cursor Agent, Droid, Gemini, KiloCode, Kiro, OpenClaw, OpenCode, Pi/OMP, Qwen, Roo Code now have `Normalizer` subclasses registered at import time. ETL pipeline now covers all 16 providers from the codeburn catalog. Beta providers stay opt-in via the existing `STACKUNDERFLOW_BETA_*` env flags — when they're enabled the matching normalizer fires automatically. Gemini + Qwen apply the canonical cached-subtraction + reasoning-fold rule that mirrors the OpenAI shape.
+- **Wave 4E — real-data ETL pipeline e2e + per-route latency regression suite.** New `tests/stackunderflow/integration/` package with two slow-marker test files: `test_etl_pipeline_e2e.py` builds a 10K-message synthetic store across 5 providers, runs backfill, validates every mart sums correctly, then hits every dashboard route asserting 200 + non-empty + <500 ms. `test_route_perf_regression.py` parametrises every dashboard route against a 100K-row synthetic marts fixture with explicit per-route latency budgets — fails CI if any route regresses. Run with `pytest -m slow`. New `[tool.pytest.ini_options]` registers the `slow` marker and adds `addopts = "-m 'not slow'"` so default `pytest tests/ -q` keeps the fast feedback loop. Latency table from the regression suite (dev box, 100K mart rows): `/api/projects` 5.8 ms, `/api/dashboard-data` 7.1 ms, `/api/cost-data` 11.9 ms, `/api/cost-data/by-provider` 1.1 ms, `/api/compare` 1.7 ms, `/api/yield` 1.2 ms, `/api/optimize` 100 ms, `/api/messages/summary` 1.6 ms.
+- **Wave 4F — ETL status badge in the dashboard header + Settings backfill button.** New `EtlStatusBadge` polls `/api/etl/status` every 10s and shows live/syncing/stale/error health with a click-through popover detailing per-mart watermarks, per-provider event counts, and watcher state. Settings page gains an "ETL pipeline" section with a "Backfill now" button — POSTs to `/api/etl/backfill` when available, else shows the equivalent CLI command. Bundle delta: +14 KB raw / +5 KB gzipped.
+
+### Test count
+1472 (start of Wave 4) → **1598 passing, 2 skipped, 11 deselected** (+126 net from Wave 4 work; ETL-specific tests +250 cumulative since start of pipeline work).
 - **Wave 4E — real-data ETL pipeline + per-route latency regression suite.** New `tests/stackunderflow/integration/` package with two slow-marker test files: `test_etl_pipeline_e2e.py` builds a 10K-message synthetic store across 5 providers, runs backfill, validates every mart sums correctly, then hits every dashboard route asserting 200 + non-empty + <500ms. `test_route_perf_regression.py` parametrises every dashboard route against a 100K-row synthetic marts fixture with explicit per-route latency budgets — fails CI if any route regresses. Run with `pytest -m slow`. New `[tool.pytest.ini_options]` section in `pyproject.toml` registers the `slow` marker and adds `addopts = "-m 'not slow'"` so the default `pytest tests/ -q` run keeps the fast feedback loop (slow tests are opt-in).
 - **Wave 3A — hot-path routes migrate to mart reads.** `/api/projects?include_stats=true`, `/api/dashboard-data`, and `/api/cost-data` (totals/by_day/by_model blocks) now read from `project_mart` + `daily_mart` instead of running per-request aggregator passes against raw `messages`. Same JSON contract; ~50× faster on the user's 28K-message project (cold 2.5–2.8s → 50ms warm). Per-session / per-command / per-tool detail blocks stay on the aggregator path until lower-grain marts ship in Wave 4.
 - **ETL foundation: usage_events fact table + 5 marts + watermarks + backfill orchestrator (Wave 1).** Lays the schema and base classes; Waves 2 (normalizers + mart builders + watcher) and 3 (route migrations) fill in the bodies. Migration v006 (the spec called it v004, but v004/v005 were taken by the synthetic-models cleanup and cursor-workspace redistribute — the migration file is renumbered to v006 and the spec doc is updated to match) adds 7 tables (`usage_events`, `daily_mart`, `session_mart`, `project_mart`, `provider_day_mart`, `model_day_mart`, `mart_watermark`) plus indexes (`idx_events_day`, `idx_events_project`, `idx_events_provider`, `idx_events_session`, `idx_events_model`, `uniq_events_msg` UNIQUE on `source_message_fk`, `idx_daily_mart_project`, `idx_session_mart_project`, `idx_session_mart_first`, `idx_provider_day_mart_day`). New `stackunderflow.etl` package: `normalize/base.py` (`Normalizer` ABC) + `normalize/__init__.py` (last-wins `register/get/all` registry), `marts/base.py` (`MartBuilder` ABC with abstract `refresh(conn, since_event_id) -> int` and concrete no-op `rebuild_from_scratch`) + `marts/__init__.py` (last-wins registry), `watermark.py` (`get_watermark` returns 0 on missing, `set_watermark` upserts with UTC ISO8601 `last_refresh_ts`, `refresh_all_marts` iterates the marts registry and persists each mart's new watermark), and `backfill.py` (`BackfillReport` dataclass with `events_inserted`, `events_skipped_duplicate`, `marts_refreshed: dict[str, int]`, `duration_seconds`; `backfill(conn, *, force=False)` orchestrator skeleton — empty-registry no-op until Wave 2 lands, `force=True` empties events + marts + watermarks). New CLI: `stackunderflow etl backfill [--force]` (no-op until normalizers register in Wave 2; reports zero counts). Migration is **additive** — does not touch existing `messages`/`sessions`/`projects` tables, all existing routes keep working unchanged. 39 new tests across `tests/stackunderflow/store/test_migration_v006.py` (12: tables exist, columns/PKs per table, indexes present, UNIQUE on `uniq_events_msg`, idempotent re-apply), `tests/stackunderflow/etl/test_registries.py` (7: register/get/all, copy semantics, last-wins overwrite for both registries), `tests/stackunderflow/etl/test_watermark.py` (9: missing→0, set/get round-trip, overwrite, ts stamping, per-mart independence, empty-registry refresh, advance + idempotent + pickup-from-existing-watermark), `tests/stackunderflow/etl/test_backfill.py` (7: empty-store report shape, idempotent re-run, `force=True` drops events + marts + watermarks, `force=True` idempotent, mart refresh runs even with empty normalizers, BackfillReport field-set is locked). Spec at `docs/specs/etl-architecture.md`.