First parallel leave-one-out batch (18 agents): quality results + isolation/audit findings

[cmungall]

First parallel leave-one-out batch — results + isolation/audit findings

We ran the curator+harmonizer agent on 18 held-out target datasets in parallel (leave-one-out; each in its own build_env sandbox), then scored outputs against the expert and audited every run's trace with the invigilator. Batch: 18 agents, ~2.0M tokens, 495 tool calls, ~21 min wall-clock. (Dataset 7 was run manually earlier → 19 folds on disk total.)

TL;DR: output quality is strong and not inflated by cheating (no agent read its held-out answer). But the run surfaced two things worth fixing before we trust the harness at scale: instruction-based isolation is leaky (agents wander out of the env), and the invigilator v1 is too noisy (mostly false positives).

---

1. Quality scorecard (vs expert)

Metric	Result
Curator include decision	18/19 (only idx 4 excluded)
Payload-file selection (exact)	18/18 of included datasets
Location-file selection (exact)	14/18 (differ: 5, 10, 23, 26)
Variable mappings	near-perfect (mostly 8/8; a few 7/8 — doc-completeness diffs)
Code ran & produced harmonized CSV	18/19 (the excluded one correctly did not)

The agent reproduced the expert's data/location file selection essentially perfectly and harmonized 18 datasets end-to-end (incl. UTM-reprojection cases, once pyproj was installed).

2. Curator disagreement — idx 4 (needs human review)

The agent excluded idx 4 (doi:10.15485/2561511, "Johnsen_Bi_2025_DAE_Manuscript_Data_Package") with four reasons — its only soil-moisture field (swc) is a depth-averaged composite, it's a denoising-autoencoder ML manuscript package, etc. The expert included it. Genuine disagreement — possibly the agent is right and the expert's inclusion is borderline. Also review the 4 location-file mismatches (5, 10, 23, 26 — several are REF_IDX-lookup datasets).

3. Isolation audit — the real headline

Raw invigilator flagged 17/18 runs. Digging into the actual violations:

• No agent read its held-out answer. Zero agents content-read a real data/gold/expert_code/harmonize_sm/dataset_NN.py. So the quality numbers are not from cheating.
• Minor real out-of-env access: 5 agents ls'd the real data/gold/ dir (saw filenames, not content); 1 (idx 3) read the head (~2 KB) of the full mapping JSON (early entries, not its own). They wandered, but didn't reach the answer.
• The other ~90% of flags were false positives (see §4).

Conclusions:

1. Instruction-based isolation is leaky at scale. The single manual ds7 run was clean, but with 18 parallel agents several wandered out of the env — reading skills/metadata from the repo-root path instead of the env copies, a few ls-ing data/gold, one heading the full mapping. None hit the answer this time, but the boundary clearly isn't respected reliably. → We should move to absence-based isolation: run each agent confined so data/gold/, the full sm_data_harmonization_mapping.json, and the complete expert code are not reachable at all. The current design keeps the real gold in the repo and builds envs under the repo (.runs/), so agents that cd to the repo root find the answers. Consider building envs outside the repo and/or running in a sandbox with a confined root.

2. The invigilator v1 needs hardening before it's a trustworthy gate — its core logic caught the real signal, but the noise makes the verdict unusable as-is.

4. Invigilator false-positive classes to fix

Concrete examples from this batch (src/folds/invigilator.py):

• Shell variables not expanded — cat "$RAW/flmd.csv", open('$d/bundle.json') → resolved literally to repo/RAW/… / repo/d/…. Should track simple VAR=... assignments and expand $VAR/${VAR}.
• Python-code fragments matched as paths — (df["dtop"]+df["dbot"])/2/100 → /2/100; min/max, site/coord, dtypes/ranges, 7/23/21 (a date). The _PATH_TOKEN regex is too eager inside heredoc/python3 -c bodies.
• Timezone strings — America/Denver, Etc/GMT+7 flagged as paths.
• System interpreter paths — /usr/bin/env. Add /usr, /bin, /opt, /etc to IGNORED_ROOTS.
• Harness-internal paths — agents reading ~/.claude/projects/…/tool-results/*.txt (their own tool results). Ignore the .claude transcript tree.
• Repo-root inputs vs env copies — agents read repo/skills/*.md and repo/data/external/ess-dive_meta/ (identical to the env copies). These are benign inputs, not answers; distinguish "input accessed outside env" (benign/warn) from "answer accessed" (violation).

After hardening, the residual real signal (ls of data/gold, head of full mapping) should be the only flags — and ideally absence-based isolation makes even those impossible.

5. Suggested next steps

• Harden the invigilator (the six FP classes above); re-audit this batch → expect a near-clean verdict with only the ls data/gold / mapping-head items.
• Absence-based isolation: build envs outside the repo and/or run agents with a confined root so answers aren't on the path.
• Human-review the idx 4 exclude and the 4 location-file mismatches.
• Output-equivalence scoring (run expert + agent code on the same raw data, diff cell-by-cell) for cell-level accuracy — the deeper quality metric.

Artifacts (local, not committed — .runs/ is gitignored)

• Per-fold agent outputs: .runs/fold_dsNN/agent_output/ (bundle.json, harmonize.py, mapping_entry.json, notes.md, harmonized CSVs).
• Agent traces: the workflow's subagents/workflows/<run>/agent-*.jsonl under the session dir.