AgentTrace v1.0.0

AgentTrace v1.0.0

Research question

Which model-agnostic behavioral telemetry signals predict eventual coding-agent
success early in a run, how much value do they add beyond task/model priors, and
how well do those signals generalize across models and agent scaffolds?

Frozen design

• 20,000 labeled Open-SWE-Traces trajectories across four model × scaffold domains
• 99,593 fixed-prefix rows at K = 3, 5, 10, 20, and 30 observable actions
• 25 frozen core models with instance_id-grouped splits
• paired task-cluster bootstrap uncertainty, calibration, low-FPR operating points,
  controls, ablations, grouped importance, and transfer tests
• frozen core anchored at commit 2779fa2; completion stages did not refit it

Headline findings

• Telemetry adds little beyond static priors through K=10, with consistent
  incremental value emerging by K=20–30.
• At K=30, combined logistic reaches ROC-AUC 0.629 and failure PR-AUC 0.705;
  HGB reaches 0.644 and 0.718.
• K=30 HGB recalls 13.8% of failures near 5% realized FPR and 22.9% near 10%.
• K=10 held-out-scaffold ROC-AUC falls to approximately 0.547–0.576.
• Grouped-label permutation and arbitrary-hash controls remain near chance;
  the stronger future-information oracle is deliberately non-deployable.

Major limitations

The study covers two model families and two agent scaffolds in benchmark-style
trajectories. Failure PR-AUC is prevalence-sensitive, low-FPR recall is limited,
and scaffold/domain transfer is weak enough to require target-domain validation.
Behavioral importance is predictive rather than causal. AgentTrace is not a
production-ready automatic stopping policy.

Reproducibility

The committed results notebook reads only the compact aggregate release and does
not rerun analysis. Full reproduction follows the frozen Kaggle CPU protocol,
dataset revision ad4805a5aa7de70d99cab0bb8f99b15304c76de0, sample seed
20260813, split seed 131313, immutable stage markers, download gates, and
checksum verification. The completed study used $0 paid compute, no GPU, no LLM
inference, and no neural training.

Release commit and assets

• Release commit: bf01763611ca0b80f4bbe52b08cf753ae3c5eaf8
• 07_aggregate.zip: 121,257 bytes
• SHA-256: b24635ff8a0cc5b5fe5981acbfaa6f69c3a81a8a4c799cb11dd03c2a83ab94fc

Only compact aggregate tables, figures, and audit metadata are attached. Raw
trajectories, predictions, fitted models, per-fit runs, and checkpoint archives are
not release assets.