Skip to content

eval: prove content-addressed catalog admission - #39

Draft
schickling-assistant wants to merge 1 commit into
compoundingtech:mainfrom
schickling-assistant:2026-07-29-content-addressed-catalog-evals
Draft

eval: prove content-addressed catalog admission#39
schickling-assistant wants to merge 1 commit into
compoundingtech:mainfrom
schickling-assistant:2026-07-29-content-addressed-catalog-evals

Conversation

@schickling-assistant

Copy link
Copy Markdown
Contributor

Why

The Agent Spec catalog proposal needs portable product-conformance evidence,
not a green reference shim. This draft adds model-free cells that invoke the
real experimental st2 API from the companion prototype.

Tracks #37 and compoundingtech/st2#52.

What

Add the hierarchical agent-spec-lifecycle-catalog-* family:

  1. prepare - exact-byte idempotent import without catalog visibility;
  2. publish-cas - explicit staging, replay, manager fencing, ABA rejection,
    and no root movement;
  3. root-atomicity - stale-root CAS, invalid-join rejection, atomic multi-seat
    admission, old/new-only reads, mixed-manager preservation, and corruption
    fail-closed.

The cells use ordinary team-less folder evals and shell/typed JSON judges. No
runner path, model seat, paid judge, or embedded catalog implementation was
added.

The VRS records typed-resource admission and public crash-boundary injection as
future discriminators; this first family does not claim those unsupported
properties and does not update the accepted AGENT-SPEC.md pin.

Exact prototype under test

Validation

Directly against that st2 binary:

  • prepare: 5 PASS / 0 FAIL
  • staging/CAS: 6 PASS / 0 FAIL
  • root atomicity: 8 PASS / 0 FAIL

Free corpus checks:

  • 77 KDL files parse;
  • 46 maintained folder evals and 9 nested catalogs validate;
  • preflight reachability, retired-surface scan, no-PII, catalog freshness,
    shell syntax, diff checks, and exact model-free harness exclusions pass.

The repository-wide harness-contract command still encounters the existing
synthetic Git identity policy rejection (seed@local versus the assistant
identity); the new three-cell exclusion coverage is independently exact.

Merge boundary

Keep this PR draft and do not promote the cells to accepted evidence until the
st2 API is accepted and the repository intentionally updates its global st2
pin. The current draft is executable incubation evidence.

Posted on behalf of @schickling
field value
agent_name co2-pine
agent_session_id cdd15622-c01a-4731-bbfe-ab8cce0cad54
agent_tool Codex CLI
agent_tool_version 0.145.0
agent_runtime Codex CLI 0.145.0
agent_model unknown
runtime_profile /nix/store/ph8rlhdj25mg71v81jsfzy6dq4xpcs9m-coding-agent-runtime-profile/share/coding-agents/profile.json
skills_manifest /nix/store/lsykz8x5481xrpbgk280xh3pypk1c5jy-agent-skills-corpus/share/agent-skills/manifest.json
worktree evals/schickling-assistant/2026-07-29-content-addressed-catalog-evals
machine dev3
tooling_profile dotfiles@3649b53

agent-session-id: cdd15622-c01a-4731-bbfe-ab8cce0cad54
agent-tool: Codex CLI
agent-tool-version: 0.145.0
agent-model: unknown
agent-runtime-profile: /nix/store/ph8rlhdj25mg71v81jsfzy6dq4xpcs9m-coding-agent-runtime-profile/share/coding-agents/profile.json
agent-skills-manifest: /nix/store/lsykz8x5481xrpbgk280xh3pypk1c5jy-agent-skills-corpus/share/agent-skills/manifest.json
tooling-profile: dotfiles@3649b53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant