Skip to content

bench: falsification fixtures for oracles + typed runner outcomes #1481

Description

@thymikee

Carved out of #1406 (closed as superseded — see its closing comment for what landed elsewhere and what was dropped). These are the two surviving work items, both deterministic PR-time quality improvements to the help-conformance bench, which is now the repo's single non-gating small-model oracle (SkillGym was removed in the #1412 descope).

1. Falsification fixtures for benchmark oracles

Every important oracle must demonstrate it can fail. Extend case/expectation definitions so each named expectation carries:

  • a minimal passing witness;
  • at least one known-bad counterexample;
  • where useful, a metamorphic variant that changes irrelevant nouns/values.

Add a completeness test that rejects a named expectation without falsification fixtures. Include regressions for swallowed lifecycle commands, unsupported flags/selectors, pseudo refs, shell operators, and invalid positional ordering.

2. Typed runner outcomes — infrastructure errors cannot enter model scoring

Replace raw-string success inference with a discriminated outcome:

type RunnerOutcome =
  | { kind: 'success'; raw: string; commands: string[] }
  | { kind: 'runner-error'; raw: string; message: string; reason: RunnerErrorReason };
  • Only success outcomes may reach command validation and scoring.
  • A result must not contain both runnerError and model-validation failures.
  • All-error aggregates render N/A, not 0/0 (0%).
  • Keep single-trial mode for exploration, but do not label it stability evidence. (The --evidence multi-trial reporting mode from Guard help-conformance correctness at production and CI boundaries #1406 is optional follow-up, on demand — not part of this issue.)

Acceptance criteria

  • Every named/important benchmark expectation has a passing witness and known-bad counterexample enforced by a completeness test.
  • Runner/API errors are structurally separate from model-plan failures and cannot affect pass-rate denominators.
  • All-infrastructure-error reports show N/A.
  • Deterministic tests only — no paid model calls in any gate.

Non-goals (decided in the #1412 descope)

Related: #1406 (superseded parent), #1433 (production positional contract, landed), #1411 (renderer-pinned bench gates, landed).

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions