WT-2-metric-extract: eval-metrics extractor (Phase 2 D2)

[yzs15]

Phase 2 #6 · 12号 §D2 · Owner: WT-2-metric-extract

Scope

• New Python package multi-agent/tools/eval/metrics/ (module eval_metrics).
• Reads observer SQLite (runs, route_reasons, capability_snapshots, task_contracts, …) → emits paper-metric CSV or JSON.
• No Go code touched. Diff scope: multi-agent/tools/eval/metrics/ + docs/specs/wt2-metric-extract.{spec,plan}.md.

Metric coverage (41)

Union of 08 号 §Metrics + §Experiments E1–E6 + 12 号 §A/§B/§C/§D + 11 号 §6 (E4 Stage-A HumanEditCount / token):

• §2.1 Lifecycle (9): feat(deploy): linux bring-up templates for observer/driver/slave + compose test #1..Implement real remote cancellation for driver cancel_task #9
• §2.2 Contracted (7): fix(driver): contract fixes from review §1.1 — preserve task_id after DelegateTask, quiet shutdown, honor SIGTERM #10..orchestrator: fanout tests regress after planner §1.5 hardening (#20/#21) #16
• §2.3 User-promoted (13): fix(planner): §1.5 prompt injection + parse fragility (#20, #21) #17..Link Codex exec sessions to parent tasks #27 + feat(eval): commit_meta collector (Phase 0 WT-0 §D6b) #40 HumanEditCount + feat(eval): §F4 ground-truth labels + JSON Schema validation #41 TokenUsage
• §2.4 Semantic (3): P2: propagate parent link end-to-end + slave serve-daemon (#24) #28..Commander: make /commander usable on phones and tablets #30
• §2.5 Overhead (9): P3: Commander cross-daemon session nesting (#24) #31..Simple list_agents query requires excessive command/tool probing in Codex driver workspace #39

Cross-section membership (per spec §3.2 subset table): TaskSuccessRate / TimeToCompletion / HumanContextSelectionCount / WrongContextFailureRate / ManualSetupStepCount / ConfigTouchCount / CapabilityReuseRate / RepeatedGenerationRate are cross-listed so --metric-set semantic|overhead|user-promoted|lifecycle all pick them up per 08:140 / 08:87 / 08:37-38.

Metrics whose upstream data source has not landed (12号 §A3 / §A4 / §A6 / §B / §C4 / §D6c / §D7 / §D8) emit null with _notes: \"upstream data missing\" and a stderr warning naming the owner. Metrics with landed upstream but empty cohort emit null with \"denominator zero\". The two reasons are a closed set — every null cell has exactly one.

CLI

eval-metrics extract \\
  --observer-db <path>          # required; SQLite (§7 (a)–(b))
  --runs-filter '<fragment>'    # optional; WHERE fragment (§7 (c))
  --format csv|json             # required; no default (silent-typo hazard)
  --out <path>                  # default stdout (§7 (d))
  --metric-set full|lifecycle|contracted|user-promoted|semantic|overhead   # default full (§7 (e))

Exit codes: 0 success / 2 validation / 3 write-time I/O.

Security (spec §7 (a)–(h))

Defense-in-depth on --runs-filter (the flagship attack: --runs-filter \"1=1; UPDATE runs SET success_oracle_result='pass'\" would silently corrupt the paper's evidence base):

Layer	Guards against
Denylist scan	;, --, /*, */, 20 DDL/DML keywords case-insensitive
Parametric extraction	Every string literal → ? placeholder
sqlglot AST parse (<27)	Structured node walk on the parsed expression
Column whitelist	Only run_id, workload_id, claim_id, experiment_id, baseline_or_ablation
Tree-shape walker	Every AND/OR/NOT/Paren leaf must be a predicate (rejects bare TRUE, run_id='x' OR TRUE)
Tautology guard	Every predicate must reference a whitelisted column (rejects 1=1)
IS-predicate ban	Rejects workload_id IS NOT NULL (schema-tautology on NOT NULL columns)
Predicate-shape enforcement	LHS bare Column; RHS bare literal/placeholder — rejects col = LOWER(col), col = col || '', col = COALESCE(col,''), col = -workload_id, col = -(-col), function-wrapped LHS

--observer-db: realpath scrub → /etc /proc /sys /dev reject → O_NOFOLLOW magic-byte probe → /proc/self/fd/<fd> handoff to sqlite3 on Linux (Codex round-6 P0 tripwire) → PRAGMA query_only=ON + read-back verification.

--out: parent realpath (must exist, no mkdir -p) → forbidden-subtree reject → O_RDONLY|O_DIRECTORY|O_NOFOLLOW dir_fd pin → dir_fd-relative fstatat/lstat symlink/overwrite refusal → dir_fd-relative O_WRONLY|O_CREAT|O_EXCL|O_NOFOLLOW open at mode 0600. Parent swap between validate and open lands the file in the ORIGINAL inode, not the attacker's replacement.

CSV formula-injection escape covers 7 prefix chars (= + - @ \\t \\r \\n), matching cmd/evalrun-export/main.go:283.

Fixtures + tests

• 3 golden fixtures (fixture_{1,2,3}.db + expected_{1,2,3}.json + build_fixture_{1,2,3}.py) exercise the full 41-metric matrix. Fixture-builder canonicalises short IDs to sha256 to satisfy evalrun/schema.go:42 ^[a-f0-9]{64}$.
• 122 pytest tests pass (17 test files); 1 perf test skipped by default (runs with -m perf or CI=true).
• Percentile math uses Fraction to avoid IEEE-754 dust (76.5 not 76.49999999999997; 3700000.0 not 3699999.999999999).

Handoff carve-out

14号 §5 handoff item 1 (internal/evalrun.NewSQLWriter Go-side wiring) is reassigned to a follow-up Go-only worktree; this Python-only worktree consumes whatever the SQLWriter writes to runs, whether via PR #53's stub CSV or the future real SQL writer.

Codex review trajectory

• Spec: 14 rounds (P0(7)→P1(6)→P1(6)→P1(5)→P1(6)→P1(5)→P1(2)→P1(3)→P1(4)→P0(1)→P0(1)→P1(3)→P1(2)→CLEAN)
• Plan: 2 rounds (P1(2)→CLEAN)
• Code: 7 rounds (P0(1)+P1(2)→P0(1)→P2(4)→P0(1)→P0(1)→P0(2)→CLEAN)

Every code-review P0 covered a real security bypass in the --runs-filter guard or --observer-db/--out TOCTOU handling; each was fixed with a targeted regression test.

Verification

cd multi-agent/tools/eval/metrics
PYTHONPATH=. python -m pytest -q             # 122 passed, 1 skipped
python -m eval_metrics extract --observer-db /tmp/empty.db --format csv | head -2   # header + 1 data row

🤖 Generated with Claude Code