Add evaluation benchmark suite

[HarperZ9]

Summary

• Add deterministic evaluation benchmark fixtures with the issue Build a LoopCompass evaluation and benchmark suite #5 baseline commit recorded.
• Add scripts/evaluate.mjs to render a Markdown metric report from synthetic or recorded receipts.
• Document the measurement-only fixture contract and wire the new tests into scripts/verify.mjs.
• Remove the verifier's shell-spawned Node path so Node 25 does not emit the child-process deprecation warning during verification.

Notes

This stays host-agnostic and does not require live provider integration. The initial fixture intentionally includes one blind-retry regression case so the report demonstrates consultation recall and terminal-outcome compliance failures without claiming live host performance.

Validation

• node --test tests/evaluation-fixtures.test.mjs tests/evaluation-report.test.mjs tests/release-tooling.test.mjs
• node scripts/evaluate.mjs --fixture fixtures/evaluation/cases.json
• node scripts/verify.mjs

Closes #5.

[HarperZ9]

Local verification receipt from the PR branch (agent/evaluation-benchmark-suite) on Windows:

• node --test tests/evaluation-fixtures.test.mjs tests/evaluation-report.test.mjs tests/release-tooling.test.mjs -> 9 pass, 0 fail
• node scripts/evaluate.mjs --fixture fixtures/evaluation/cases.json -> rendered loopcompass-core-eval-v1 report with baseline d7879fec762322ae658603104c7c334ade6ba43f
• node scripts/verify.mjs -> 51 pass, 0 fail; release inventory validation and example redaction also passed

The benchmark remains host-agnostic and no live provider credentials were used.

[HarperZ9]

Pushed a follow-up tightening pass on the benchmark scope.

What changed since the initial PR:

• Expanded the deterministic fixture from 6 to 10 cases.
• Added explicit scope dimensions for Codex, Claude, Grok CLI, parent agents, read-only subagent handoff, missing-skill fallback, and missing project instructions.
• Split host enforcement quality from skill decision quality in the generated report, so missed consultations are measurable without blaming the LoopCompass decision path when the skill was never invoked.
• Added repeated-failure reduction and time-to-verified-normal-path metrics while keeping the no-live-provider, measurement-only boundary.

Local verification on Windows from 454ceb2380f6a98667628138e17d1bec5308f667:

• node --test tests/evaluation-fixtures.test.mjs tests/evaluation-report.test.mjs tests/release-tooling.test.mjs -> 10 pass, 0 fail
• node scripts/evaluate.mjs --fixture fixtures/evaluation/cases.json -> renders 10-case loopcompass-core-eval-v1 report with baseline d7879fec762322ae658603104c7c334ade6ba43f
• node scripts/verify.mjs -> 52 pass, 0 fail; release inventory validation and example redaction passed
• git diff --check -> clean

GitHub currently reports no configured checks on this branch.

[adammmmmm]

Thanks again for putting this together. Directionally, this is exactly the shape I was hoping for: deterministic, credential-free, host-agnostic, and kept out of the LoopCompass runtime itself.

I had the implementation pressure-tested by three independent skeptic reviews. They all landed in roughly the same place: the infrastructure is solid, but a few of the current metrics can be read as stronger evidence than the fixtures actually support.

The biggest issue is classification accuracy in scripts/evaluate.mjs. It scores every case, including cases where LoopCompass was never consulted. That lets cases 005 and 010 count as correct classifications even though no skill decision occurred, which is how the report gets to 10/10.

There is also a contradiction in case 010. The scenario says the same failure was retried without consulting project memory and the attempt count did not improve, but both the receipt and expectation record blind_retry: false. I think the recorded value should be true and the desired value should remain false, so the report captures the regression.

Before merging, could you please tighten up the following?

1. Restrict classification accuracy to consulted cases, or rename it so it is clearly just receipt-label agreement.
2. Correct the blind-retry values for case 010.
3. Validate the required fixture fields and enums before scoring. Missing values currently have the potential to compare equal and count as passes.
4. Either require consultation for repeated-failure reduction and time-to-normal-path credit, or rename those metrics so they do not imply LoopCompass caused the improvement.
5. Put a clear synthetic fixtures, not live host evidence notice directly in the generated report, not only in the surrounding documentation.
6. Add focused regression coverage for an incomplete fixture, a consulted-but-wrong decision, and either a false trigger or stale-rejection miss.

I am not asking for live-provider integration, more infrastructure, or a broader framework here. The lean implementation strategy is right. This is mainly about making sure the scorer cannot accidentally produce authoritative-looking results from incomplete or self-confirming fixture data.

Once those points are addressed, I think this will be a strong first measurement layer for LoopCompass.

[HarperZ9]

Addressed the requested scorer tightening in 4ab0587.

What changed:

1. Replaced all-case Classification accuracy with Classification accuracy when consulted, so unconsulted rows no longer make classification evidence look stronger than it is.
2. Corrected case 010: observed receipt.blind_retry is now true, expected remains false, and the report records it as a blind-retry failure.
3. Added fixture validation before scoring required fields/enums, including receipt/expected booleans, classifications, terminal outcomes, repeated-failure counters, nullable normal-path steps, and scope metadata. Incomplete fixtures now fail closed instead of letting missing values compare equal.
4. Repeated-failure reduction and time-to-verified-normal-path credit now require actual consultation.
5. The generated report now includes Synthetic fixtures only; not live host evidence. directly under the title.
6. Added regression coverage for incomplete fixtures, consulted-but-wrong decisions, unconsulted improvement credit, and false triggers.

Fresh local validation on Windows:

• node --test tests/evaluation-fixtures.test.mjs tests/evaluation-report.test.mjs tests/release-tooling.test.mjs -> 14 pass, 0 fail
• node scripts/evaluate.mjs --fixture fixtures/evaluation/cases.json -> generated the updated 10-case report
• node scripts/verify.mjs -> 56 pass, 0 fail; release inventory validation and example redaction passed
• git diff --check -> clean

[adammmmmm]

Thanks for tightening this up. The original review items all look addressed, and the full test suite is green.

I think we’re very close. Before merging, could you make one final focused pass on three report-integrity issues?

1. Require attempt and step counts to be nonnegative integers. Right now negative or fractional values pass validation, so values like before: 0, after: -1, or steps: -1 could incorrectly receive improvement credit.

2. Make the report watermark reflect the actual receipt type. The evaluator accepts both synthetic and recorded receipts, but every report currently says Synthetic fixtures only. Recorded receipts should be labeled accurately while still making clear that they are not live-host evidence unless they came from an explicit live run protocol.

3. Require scope.host and receipt.host to match, unless there is an intentional and clearly reported reason for them to differ. Coverage currently uses one field while performance reporting uses the other, so a mismatched fixture could attribute results to the wrong host.

No broader changes needed. Once those are handled, I think we’re good to merge.

[HarperZ9]

Implemented the final report-integrity pass in 33686c1.

1. Attempt counts, step counts, and step budgets now require nonnegative integers; nullable normal-path steps still permit null.
2. The watermark now derives from the receipt types actually present (synthetic, recorded, or mixed) and always preserves the non-live-evidence caveat because schema 1 has no explicit live-run protocol contract.
3. receipt.host must match scope.host. There was no existing mismatch-justification contract, so mismatches fail closed before scoring.

Regression coverage now includes negative and fractional counters, recorded and mixed watermarks, and host mismatches. Documentation describes the same contract.

Test output from the pushed head:

• node --test tests/evaluation-fixtures.test.mjs tests/evaluation-report.test.mjs tests/release-tooling.test.mjs -> 19 pass, 0 fail
• node scripts/evaluate.mjs --fixture fixtures/evaluation/cases.json -> renders the receipt-type-aware watermark
• node scripts/verify.mjs -> 61 pass, 0 fail; release inventory and example redaction checks passed
• git diff --check -> clean

No broader changes were made. Ready for final re-review.

[adammmmmm]

The final report-integrity pass addresses the requested validation, provenance, and host-attribution issues. Focused and full verification are green. Ready to merge.