Skip to content

Release v3.98.0

Choose a tag to compare

@github-actions github-actions released this 25 Oct 04:43
d002f53

Added

  • cratis ai test runs opt-in AI conformance fixtures against installed harnesses (Claude Code and Codex). Each fixture × harness gets a fresh temporary workspace that installs only that fixture's profile and language slice, so the invoking repository is never changed. Fixtures come from the corpus Evaluations/conformance folder, or from --fixtures. (part of #204)
  • Fixture checks are deterministic: required and forbidden files, per-file regex matches, and optional commands. They can run against the installed tree before the agent starts or against the result. A glob that matches no files fails, and no check can reach outside the workspace.
  • Results keep the evidence levels apart: installed instruction files with their hashes, whether the agent was observed reading them (unknown unless the harness reports a read), and the check outcomes. A missing harness or missing credentials is reported as skipped with a reason, never as a pass. --output json gives machine-readable results for CI.
  • Every run is bounded. A timeout kills the whole process tree, Claude runs with a spending limit (reported as not_enforced for Codex, which cannot enforce one), and output capture is capped. Transcripts are kept only with --keep-transcripts, inside the temporary output, with secrets redacted. See Test AI conformance.