Skip to content

v0.19.0 — Measure the value, not just the firing

Choose a tag to compare

View all tags
@lancegui lancegui released this 12 Jun 18:59
· 113 commits to main since this release
v0.19.0
90ccb6a

The family's evals tested whether skills trigger; nothing tested whether following them catches anything. This release measures both halves.

New

  • Behavioral benchmark (evals/behavioral/ + scripts/run-behavioral-eval.py): 9 tasks, each with one planted silent failure (fan-out join, silently filtered rollup, cents-vs-dollars, top-coded missingness, train/test overlap, post-treatment "control", diverging pre-trends, spatial silent drop, non-identified elasticity). Deterministic generator (rubric numbers computed from the data), claude -p A/B arms under an isolated CLAUDE_CONFIG_DIR (baseline verified plugin-free), LLM-graded. First calibration run (sonnet-4-6): card 8/9 vs baseline 8/9 — near-ceiling both arms; the one clean differential was train/test overlap, caught only with the card. Honest analysis + v1 hardening plan: docs/2026-06-12-behavioral-benchmark-v0.md.
  • Trigger CI (scripts/eval-triggers.py): the eval corpus through the real prompt-router vs a committed baseline (precision 145/145, recall 54/127 — a backstop by design); fails on regressions. --live --competitors tests description matching against the overlapping superpowers skills — the "brainstorming steals the trigger" failure mode, now measurable.
  • data-contracts ships code: references/contract-helpers.md — assert_join / reconcile / na_audit / freeze_baseline preludes in Python, R, Julia, and Stata.
  • Stale-plugin warning in hooks/session-start (counters claude-code#52218 — the mechanism behind "the skills never triggered" on a stale session).

Improved

  • Lessons loop closed: wrong-number-debugging and analysis-review end by logging the failure class to the project's docs/LESSONS.md.
  • Always-on card diet: 1,258 → 970 words (−23%), zero rules dropped; AGENTS.md inherits via symlink.
  • question-framing → data-viz deliverables: a map/figure/dashboard built from a dataset now triggers framing across all five surfaces (fixes the real-session leaflet-map miss).
  • Ecosystem survey: docs/2026-06-12-evolution-candidates.md (ranked adoption candidates).
  • CHANGELOG backfilled 0.11.0–0.18.1; tags + releases created for all skipped versions.

Update: /plugin update causal-powers@causal-powers, then restart Claude Code — and from now on the plugin will tell you itself when it's stale.

Assets 2
Loading