Proposal: support external agent eval fixture packs

[bennewell35]

Problem

Small Harness has a useful agent eval path (small-harness --eval ..., /eval agent ..., and /play ...), but eval fixtures are currently registered in Rust source via the built-in fixture list. That makes custom or domain-specific eval packs awkward: adding one requires changing and rebuilding Small Harness itself instead of dropping a fixture into a project or sharing a small fixture directory.

This limits use cases like:

• comparing local models against the same project-specific tasks
• shipping reusable eval packs alongside a repo
• testing domain-specific agent behavior without forking Small Harness
• using /play-style sandboxes for non-built-in tasks

Proposal

Add support for external AgentEvalFixture definitions loaded from a file or project directory while keeping the current built-ins unchanged.

A minimal first version could support:

small-harness --eval ./evals/fixtures/fix-readme-badge.json --model qwen2.5:7b

and optionally later:

/eval agent ./evals/fixtures/fix-readme-badge.json
/eval agent project

Suggested fixture shape can mirror the existing serialized Rust structs:

{
  "id": "fix-readme-badge",
  "prompt": "Update the README version badge to match Cargo.toml.",
  "workspace": "workspaces/fix-readme-badge",
  "checks": [
    { "type": "fileContains", "path": "README.md", "needle": "version-1.0.4" },
    { "type": "testsPass" }
  ]
}

Design constraints

• Preserve all existing built-in fixtures and fixture IDs.
• Keep external fixtures data-only at first; no arbitrary commands in fixture files.
• Resolve fixture workspaces relative to the fixture file or fixture-pack root.
• Reject workspace paths that escape the fixture pack/root.
• Reuse existing check types initially (testsPass, fileContains, gitClean, toolUsed, assistantMentions).
• Produce clear errors for unknown fixture paths, malformed JSON, unknown checks, or missing workspaces.

Acceptance criteria

• small-harness --eval <builtin-id> continues to work unchanged.
• small-harness --eval <path-to-fixture.json> runs an external fixture.
• External fixture workspace copy behavior matches built-ins.
• External fixture loading has unit tests for happy path, malformed fixture, missing workspace, and path traversal rejection.
• README/Quickstart document a small external fixture example.

This would make Small Harness more useful as a general local-agent benchmark harness without forcing every eval scenario into the main repository.

[HarperZ9]

The external fixture-pack direction seems especially useful if the pack format also preserves why a result passed, not just whether the harness returned green.

One small addition I would consider for the first version: let a fixture pack optionally emit a run receipt beside the normal eval result. Nothing executable needs to live in the fixture file; it can still stay data-only. The receipt is just the replayable join between:

• fixture pack id and version
• fixture file hash
• workspace copy/root hash, if cheap to compute
• model/provider/config label
• check ids and structured check outcomes
• tool/action ids observed during the run
• artifact refs for stdout, diffs, or test output

That gives downstream users a way to compare fixture packs across machines without trusting a free-form transcript. It also makes failed external packs easier to share: "this failed because fileContains:readme-version failed against artifact X" is much easier to debug than "agent failed the task".

For the path rules, I would keep the fixture root as the authority boundary and make every external ref resolve through it:

{
  "packId": "readme-maintenance",
  "packVersion": "0.1.0",
  "fixtures": ["fixtures/fix-readme-badge.json"],
  "workspaceRoot": "workspaces",
  "receipt": { "mode": "jsonl", "includeDiffStats": true }
}

Then the traversal tests can cover both workspace and artifact/receipt paths with the same resolver. I have been building Project Telos around this kind of local receipt loop for agent work, so I am biased, but this issue looks like the right place to make fixture packs portable without making them opaque: https://harperz9.github.io/field-guide.html

[bennewell35]

This receipt direction makes sense to me, especially the part about preserving why a fixture passed/failed without relying on a free-form transcript. I would still keep the first implementation deliberately small so external fixtures do not turn into a general artifact framework on day one.

A scoped first slice could be:

1. External fixture loading

   • small-harness --eval <fixture.json> resolves a data-only fixture from disk.
   • Built-in fixture IDs keep working unchanged.
   • workspace resolves relative to the fixture file or pack root and cannot escape that root.

2. Optional pack metadata

   • Support a small manifest shape like packId, packVersion, fixtures, and workspaceRoot.
   • Treat the manifest as metadata/routing only: no commands, no executable hooks.

3. Minimal run receipt

   • Emit alongside the normal eval result, probably in the same eval output directory at first.
   • Include only deterministic fields that already exist or are cheap to compute:
     • fixture id
     • optional pack id/version
     • fixture file hash
     • backend/model label
     • check ids/types and structured check outcomes
     • observed tool call names
     • transcript path / eval JSON path
   • Leave workspace hashing, diff artifact refs, stdout artifact refs, and richer artifact indexing for a follow-up once the core loader is merged.

That keeps the first PR reviewable: path-safe fixture loading plus enough receipt metadata to make external eval runs comparable and debuggable. The traversal tests can use one resolver for fixture, workspace, and receipt paths, as you suggested.

If this direction sounds right, I can take a first pass at the narrow loader + minimal receipt slice rather than trying to solve full fixture-pack provenance in one PR.

[HarperZ9]

Yes, that first slice sounds like the right shape.

The acceptance boundary I would keep tight is:

• external fixture loading is data-only, with built-in fixture IDs unchanged;
• fixture, workspace, and receipt paths all go through the same root-bounded resolver;
• pack metadata is routing metadata only, with no commands or executable hooks;
• the first receipt is deliberately small: fixture id, optional pack id/version, fixture file hash, backend/model label, check ids/types/outcomes, observed tool call names, and transcript/eval JSON path.

I would also make the omitted pieces explicit rather than implicit. For example, workspace hash, stdout/stderr artifacts, diffs, and richer artifact indexes can be not_captured or simply absent in v1, but the run should not look like it made a stronger provenance claim than it did.

The traversal tests can probably cover most of the risk surface: fixture path escapes, workspace path escapes, receipt output path escapes, symlink edge cases if supported, and built-in fixture IDs still resolving through the old path.

If useful, I can put together a tiny Telos-style fixture pack for dogfooding: one passing check, one failing structured check, and one intentionally denied out-of-root workspace reference. That would pressure-test the loader and minimal receipt without pulling the PR into full artifact provenance yet.

[bennewell35]

That fixture pack would be useful. If you can share it, I can use it as a dogfood case while keeping the first PR to this boundary: data-only external loading, one root-bounded resolver, built-ins unchanged, routing-only pack metadata, and a minimal receipt that is explicit about what is not captured yet.

[HarperZ9]

Yes. I put together a deliberately small Telos-side dogfood pack for this boundary.

Local pack path in the Telos workspace:

demo/integrations/smallharness-dogfood-pack/

What it contains:

• pack.json: routing-only metadata with packId, packVersion, workspaceRoot, builtInFixtureCompatibility: "unchanged", and fixture entries.
• fixtures/pass-basic.json: a pass case for a receipt with verification.verdict = MATCH.
• fixtures/fail-structured.json: a structured fail case where the receipt exists but verdict is DRIFT while the check expects MATCH.
• fixtures/deny-out-of-root.json: a resolver negative case with workspace: "../outside-smallharness-pack" and expected path_escape_denied.
• receipts/*.receipt.json: minimal first-slice receipts with fixture id, pack id/version, fixture file hash, backend/model label, check ids/types/outcomes, observed tool-call names, transcript/eval JSON refs, and explicit notCapturedYet.

The pack-level shape is:

{
  "schema": "project-telos.smallharness-fixture-pack/v0",
  "packId": "project-telos.receipt-boundary-dogfood",
  "packVersion": "2026.06.28",
  "workspaceRoot": "workspace",
  "builtInFixtureCompatibility": "unchanged",
  "routingOnly": true,
  "noExecutableHooks": true,
  "fixtures": [
    {
      "id": "telos.receipt.pass_basic",
      "path": "fixtures/pass-basic.json",
      "expectedOutcome": "pass",
      "receipt": "receipts/pass-basic.receipt.json"
    },
    {
      "id": "telos.receipt.fail_structured",
      "path": "fixtures/fail-structured.json",
      "expectedOutcome": "fail",
      "receipt": "receipts/fail-structured.receipt.json"
    },
    {
      "id": "telos.resolver.deny_out_of_root",
      "path": "fixtures/deny-out-of-root.json",
      "expectedOutcome": "block",
      "receipt": "receipts/deny-out-of-root.receipt.json"
    }
  ]
}

The receipt intentionally names what is not captured yet:

"notCapturedYet": [
  "workspaceHash",
  "diffArtifactRefs",
  "stdoutArtifactRefs",
  "stderrArtifactRefs",
  "artifactIndex"
]

I also added a local verifier, demo/smallharness-fixture-pack.test.mjs, that asserts:

• pack and fixtures contain no commands, scripts, hooks, setup/teardown, shell, or exec fields;
• fixture, workspace, receipt, transcript, and eval refs go through one root-bounded resolver;
• non-negative workspaces stay under the declared workspaceRoot;
• the out-of-root fixture fails as path_escape_denied;
• receipt fixtureFileHash values bind to the exact fixture bytes;
• built-in fixture IDs are explicitly left unchanged.

Local verification:

• red: node demo\smallharness-fixture-pack.test.mjs failed first on missing pack.json, then on placeholder fixture hashes.
• green: node demo\smallharness-fixture-pack.test.mjs exited 0 after adding the pack and binding hashes.
• adjacent checks: node demo\action-receipt.test.mjs, node demo\context-envelope.test.mjs, node demo\integrations.test.mjs, and git diff --check all exited 0.

This is not a SmallHarness PR or a claim that the current loader already accepts this exact schema. It is a copyable dogfood fixture shape for the first slice you described. If you want field names adjusted to match the loader branch, I can keep the pack data-only and rename the surface without changing the test intent.

[HarperZ9]

Quick follow-up with public links instead of only local paths.

Branch/commit:

https://github.com/HarperZ9/telos/tree/outreach-fixture-validation-2026-06-28

SmallHarness dogfood pack:

https://github.com/HarperZ9/telos/blob/outreach-fixture-validation-2026-06-28/demo/integrations/smallharness-dogfood-pack/pack.json

Local verifier:

https://github.com/HarperZ9/telos/blob/outreach-fixture-validation-2026-06-28/demo/smallharness-fixture-pack.test.mjs

The branch also includes the pass/fail/out-of-root fixture files and the minimal receipts under:

demo/integrations/smallharness-dogfood-pack/

Boundary remains the same: this is a Telos-side dogfood fixture shape, not a claim that SmallHarness already accepts this schema. It is meant to be easy to rename/trim while preserving the first-slice test intent.

[HarperZ9]

Opened a narrow implementation PR for the first slice here: #13

Scope is intentionally small: path-loaded data-only fixture JSON, built-in IDs unchanged, workspace refs resolved relative to the fixture file, and traversal rejection for workspace/fileContains refs. I left pack manifests and richer receipt metadata for a follow-up so this stays reviewable.