You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This commit was created on GitHub.com and signed with GitHub’s verified signature.
Added
cratis ai test runs opt-in AI conformance fixtures against installed harnesses (Claude Code and Codex). Each fixture × harness gets a fresh temporary workspace that installs only that fixture's profile and language slice, so the invoking repository is never changed. Fixtures come from the corpus Evaluations/conformance folder, or from --fixtures. (part of #204)
Fixture checks are deterministic: required and forbidden files, per-file regex matches, and optional commands. They can run against the installed tree before the agent starts or against the result. A glob that matches no files fails, and no check can reach outside the workspace.
Results keep the evidence levels apart: installed instruction files with their hashes, whether the agent was observed reading them (unknown unless the harness reports a read), and the check outcomes. A missing harness or missing credentials is reported as skipped with a reason, never as a pass. --output json gives machine-readable results for CI.
Every run is bounded. A timeout kills the whole process tree, Claude runs with a spending limit (reported as not_enforced for Codex, which cannot enforce one), and output capture is capped. Transcripts are kept only with --keep-transcripts, inside the temporary output, with secrets redacted. See Test AI conformance.