Agentic Evidence Lab v0.1.0-alpha.5

Agentic Evidence Lab v0.1.0-alpha.5

Alpha.5 makes the existing evidence easier to inspect without changing what
the historical studies proved.

What changed

• ael results build deterministically generates the root Results Index,
  three human result cards, and a machine-readable index from an explicit,
  hash-bound public-results profile.
• ael results check fails when committed projections drift from their source
  receipts or when a selected claim exceeds the receipt's evidence ceiling.
• Every card states the exact public verification command, what that command
  proves, which material is unavailable, and which historical lifecycle facts
  were never declared.
• CI protects the released PBT v2 freeze/result bytes and validates the actual
  wheel and source archive contents and version metadata.
• The PBT v2 preregistration tag is required to resolve to the exact commit
  recorded by the frozen result bundle.

Evidence boundary

This release adds no model calls and no new effect estimate. It does not modify
the five Contract v0 schemas, old receipts, the receipt renderer, or the frozen
alpha.3/alpha.4 PBT evidence. The cards are publication projections, not new
evidence documents.

evidence_level, claim status, reproducibility, independence, current
publication, freshness, action, and downstream outcome remain separate. A
successful public audit can verify an evidence graph and Git artifact ordering;
it is not a model rerun, independent replication, or production-outcome proof.

Verify

From a full Git checkout:

uv sync --locked --all-groups
uv run ael results check studies/public-results.json --require-git-proof
uv run python tools/check_frozen_artifacts.py
uv run python -m unittest discover -s tests -v
uv run python tools/release_check.py

The exact release commit, CI state, built-archive checksums, and release manifest
belong to the GitHub release. The commands above do not rerun historical model
calls or disclose private holdouts.

Historical note: the alpha.3 checksum file used path-prefixed entries. Alpha.5
release checksums use flat asset filenames so a normal GitHub release download
can be verified directly.

Agentic Evidence Lab v0.1.0-alpha.4

Agentic Evidence Lab v0.1.0-alpha.4

This alpha makes a completed frozen study auditable as one bundle. It does not
add a new scored experiment or change the negative Property-based Testing v2
result published in alpha.3.

Included

• ael study audit for the public freeze, terminal decision, Contract v0
  graph, exact schedule, public decision counts, receipt coverage, private-pack
  digests, and optional retained private-pack verification;
• an explicit pbt-v2 adapter that recomputes this study's public counts and
  terminal outcome without presenting PBT-specific semantics as universal;
• Git preregistration proof that requires the recorded commit to contain the
  exact freeze bytes, precede the current checkout, and not yet contain the
  terminal decision;
• a machine-readable ael.study-audit/0.1 summary via --json-output;
• ael study activation-check for stronger Codex JSONL activation evidence;
• CI and clean-wheel gates for the complete published PBT v2 bundle;
• an updated locked Ruff maintenance range through 0.16.x.

What the audit proves

For Property-based Testing v2, the public audit loads 12 Contract v0 documents,
checks eight terminal run records and 88 measurements, and reconstructs the
published continuation counts. It verifies the immutable freeze bytes at the
recorded preregistration commit and verifies that the terminal decision did not
exist there.

When the retained private roots are supplied explicitly, their current tree
digests also match the frozen screening and confirmation hashes. Those private
bytes remain outside Git.

Activation evidence

The new event parser rejects mere path mentions, in-progress commands, failed
commands, and empty output. A skill is operationally activated only when a
completed exit-zero Codex command retrieves non-empty content from the exact
installed SKILL.md path. Rechecking the four retained PBT v2 treatment event
streams with this stricter rule still yields 4/4 activations. This local
recheck does not alter the frozen alpha.3 evidence artifacts.

Limits

The audit does not expose or reconstruct private tasks, raw events, candidates,
or evaluator outputs. It verifies their declared digests and independently
recomputes public counts, but it is not an independent replication and does not
establish transfer or production impact.

The package version is 0.1.0a4; the Git tag is v0.1.0-alpha.4. Contract v0,
the study-freeze contract, and the new audit-summary shape remain pre-stable.

Agentic Evidence Lab v0.1.0-alpha.3

Agentic Evidence Lab v0.1.0-alpha.3

This alpha publishes AEL's first preregistered third-party skill-effectiveness
pilot. The result is negative and bounded: adding the exact pinned Trail of
Bits property-based-testing skill did not improve hidden acceptance on the
eight repeat-1 cells, and the frozen critical-failure rule stopped the study.

Included

• a public freeze created before scored calls, bound to private task-pack
  composites, exact randomized schedules, runner, analysis and decision code,
  prompt, runtime images, budgets, and sequential stop rules;
• four private screening tasks across serialization-roundtrip and
  normalization-idempotence defect families plus two untouched confirmation
  tasks that remained locked;
• eight valid Codex run records, 88 evaluator-owned measurements, a frozen
  continuation decision, and a machine-readable evidence receipt;
• generic fail-closed study-freeze validation and a study-local sanitized
  evidence materializer;
• exact-SHA CI and an immutable property-based-testing-v2-freeze tag proving
  that preregistration preceded scored execution.

Result and decision

The four baseline/treatment matched pairs all tied on binary hidden acceptance:
both conditions passed two tasks and failed two. The treatment skill activated
in all four cells. Both conditions also produced two added-test incompatibility
failures; treatment therefore had two critical failures under the frozen gate.

The continuation outcome is reject_all_critical_failure. Repeat 2 and the
untouched confirmation pack remained locked. The supported decision is to
reject this exact skill injection for this exact pilot surface, not to reject
property-based testing, the upstream skill, Codex, or skills in general.

Treatment used 47,330 generated-work tokens and 857,633 ms total wall time;
baseline used 37,341 tokens and 752,432 ms. With four cells per condition and no
quality difference, these are descriptive pilot costs, not stable estimates.

Research integrity

The public preregistration commit is
610f0d9e1e19d9c89dd6beba8fab7900222df5dd. GitHub CI run 31604199862
passed before the freeze tag and before the first scored call. Private tasks,
raw events, candidate workspaces, and evaluator outputs remain outside Git;
public artifacts expose allowlisted measurements and immutable hashes.

Kizz authored the tasks, operated the runner, and evaluated the result. This is
maintainer-evaluated evidence, not independent replication.

Compatibility

The package version is 0.1.0a3; the Git tag is v0.1.0-alpha.3. Contract v0
and the study-freeze contract remain pre-stable.

Verify the release

Run the Python 3.11-3.13 test matrix, Contract v0 validation, source-lock and
freeze checks, public release scan, package build, clean-wheel validation, and
Docker isolation smoke against the exact release SHA. Distribution assets
include SHA-256 checksums.

Agentic Evidence Lab v0.1.0-alpha.2

Agentic Evidence Lab v0.1.0-alpha.2

This alpha publishes Agent Skills Season 1 activation calibration: a bounded,
machine-readable answer to whether twelve exact public skill snapshots can run
and activate in the controlled Codex adapter. It is not an effectiveness
leaderboard.

Included

• ten independent Season 1 protocols and exact source locks for twelve public
  skill trees from four upstream repositories;
• a healthy ten-task public calibration pack, revisioned through two retained
  benchmark invalidations and one format-only transition;
• 22 valid final-revision Codex run records, ten measurement sets, and ten
  evidence receipts;
• explicit activation evidence for ten treatment skills and inconclusive
  activation for Anthropic mcp-builder and webapp-testing;
• source-lock validation and non-executing caller-checkout verification;
• exact study-revision resolution in the evidence validator;
• a public first-wave screening boundary that keeps private screening and
  confirmation packs outside the Git worktree;
• a release-canary guard against accidental private-pack publication.

Decision

The runner and evidence pipeline are ready for study-specific screening design.
The ten activated snapshots are execution-compatible candidates only. MCP and
webapp activation redesign is deferred to their later study milestones and does
not block the declared three-study first wave.

No evidence in this release establishes that a skill improves correctness,
debugging, testing, security, design, cost, latency, transfer, or production
outcomes. Shared acceptance on one public mechanics task cannot establish
equivalence. No universal or cross-study ranking is published.

Integrity corrections

Activation found two benchmark defects that static pack-health checks missed:

1. an underdetermined truthful-completion state prefix;
2. an external-resource false positive on an inline SVG namespace.

The affected raw runs remain private and content-addressed; public invalidation
records explain their exclusion. Final evidence uses only corrected revisions.

Security boundary

The Codex process can read the maintainer's reusable ChatGPT credential. The
owner accepted that property for the exact pinned and reviewed Season 1
snapshots in maintainer-controlled fixtures. This is not a public submission
service: arbitrary or changed third-party content remains blocked pending its
own pin, review, and explicit acceptance or a stronger credential boundary.

Compatibility

The package version is 0.1.0a2; the Git tag is v0.1.0-alpha.2. Contract v0
remains pre-stable. Study references now resolve by exact ID and revision;
duplicate identical revisions fail validation.

Verify the release

Run the Python 3.11–3.13 CI matrix, public evidence validation, source-lock and
manifest checks, public calibration reproduction, Docker isolation smoke,
release-tree scan, package build, and clean-wheel validation against the exact
release SHA. Distribution assets include SHA-256 checksums.

Agentic Evidence Lab v0.1.0-alpha.1

Agentic Evidence Lab v0.1.0-alpha.1

This is the first public alpha of Kizz Agentic Evidence Lab: a file-first method
for testing versioned changes to agent behavior without collapsing the result
into a universal leaderboard score.

Included

• five Contract v0 JSON document types and cross-document validation;
• deterministic receipt rendering and study-design calibration;
• a sanitized Council Generation 1 study with twelve run records;
• a six-cell Codex coding-skill calibration with an explicit inconclusive
  treatment result;
• an offline Docker runner and a controlled-egress Codex adapter;
• task-pack health and hidden-acceptance evaluation helpers;
• security, governance, contribution, reproducibility, and schema policies.

What the evidence supports

The Council Generation 1 mapping supports a narrow local workflow decision and
records a historical execution-conformance failure. The Codex calibration
supports that the runner captured six stable cells and activated the treatment
skill as intended. It does not show that the skill improved correctness: all
six cells passed, creating a ceiling effect.

Important security limit

Hosted Codex execution is only for maintainer-controlled inputs. The agent
process can read its reusable credential; the egress allowlist is not a
credential broker. The CLI requires --trusted-input-only, and third-party
submissions must use the offline path until a safer credential design exists.

Compatibility

The package version is 0.1.0a1; the Git tag is v0.1.0-alpha.1. Contract v0
is pre-stable and may change incompatibly before 1.0.

Verify the release

Download the attached wheel or source distribution and verify it against
SHA256SUMS. The same revision is validated by the repository CI matrix on
Python 3.11, 3.12, and 3.13 plus a Docker isolation smoke job.