Agentic Evidence Lab v0.1.0-alpha.3

Agentic Evidence Lab v0.1.0-alpha.3

This alpha publishes AEL's first preregistered third-party skill-effectiveness
pilot. The result is negative and bounded: adding the exact pinned Trail of
Bits property-based-testing skill did not improve hidden acceptance on the
eight repeat-1 cells, and the frozen critical-failure rule stopped the study.

Included

• a public freeze created before scored calls, bound to private task-pack
  composites, exact randomized schedules, runner, analysis and decision code,
  prompt, runtime images, budgets, and sequential stop rules;
• four private screening tasks across serialization-roundtrip and
  normalization-idempotence defect families plus two untouched confirmation
  tasks that remained locked;
• eight valid Codex run records, 88 evaluator-owned measurements, a frozen
  continuation decision, and a machine-readable evidence receipt;
• generic fail-closed study-freeze validation and a study-local sanitized
  evidence materializer;
• exact-SHA CI and an immutable property-based-testing-v2-freeze tag proving
  that preregistration preceded scored execution.

Result and decision

The four baseline/treatment matched pairs all tied on binary hidden acceptance:
both conditions passed two tasks and failed two. The treatment skill activated
in all four cells. Both conditions also produced two added-test incompatibility
failures; treatment therefore had two critical failures under the frozen gate.

The continuation outcome is reject_all_critical_failure. Repeat 2 and the
untouched confirmation pack remained locked. The supported decision is to
reject this exact skill injection for this exact pilot surface, not to reject
property-based testing, the upstream skill, Codex, or skills in general.

Treatment used 47,330 generated-work tokens and 857,633 ms total wall time;
baseline used 37,341 tokens and 752,432 ms. With four cells per condition and no
quality difference, these are descriptive pilot costs, not stable estimates.

Research integrity

The public preregistration commit is
610f0d9e1e19d9c89dd6beba8fab7900222df5dd. GitHub CI run 31604199862
passed before the freeze tag and before the first scored call. Private tasks,
raw events, candidate workspaces, and evaluator outputs remain outside Git;
public artifacts expose allowlisted measurements and immutable hashes.

Kizz authored the tasks, operated the runner, and evaluated the result. This is
maintainer-evaluated evidence, not independent replication.

Compatibility

The package version is 0.1.0a3; the Git tag is v0.1.0-alpha.3. Contract v0
and the study-freeze contract remain pre-stable.

Verify the release

Run the Python 3.11-3.13 test matrix, Contract v0 validation, source-lock and
freeze checks, public release scan, package build, clean-wheel validation, and
Docker isolation smoke against the exact release SHA. Distribution assets
include SHA-256 checksums.