Skip to content

Evaris v0.2.0

Choose a tag to compare

@akkaraponph akkaraponph released this 15 Nov 14:57
· 23 commits to main since this release

Evaris 0.2.0 shows the evaluation workflow on real external agent sessions. The CLI, commands and stored schemas are unchanged from 0.1.0.

  • Runbook for evaluating a real external agent: docs/pilot.md (validate, prepare, run externally, import observed metadata, evaluate, report, compare), with lessons for the Claude Code CLI host: permission mode, scratchpad location, sources for observed metadata and instruction-loading detection. Documented commands use one binary path.
  • A preregistered process pilot, kept private with its evidence: 12 real Claude Code CLI sessions in 3 trials (2 original Go cases × baseline/specialist), with every observed field taken from the host transcript. One case separated the arms descriptively (baseline failed one stated rule in 2 of 3 trials). Every comparison was NOT_COMPARABLE because specialists read the instruction file through a shell command that the preregistered loading rule did not count. This is not evidence of agent or instruction effectiveness.
  • An independent AI operator session reproduced all 12 evaluations and 3 comparisons from the retained evidence. No second person reproduced the grading, and the summary was reviewed by an independent AI session.

Install

Download the binary for Linux or macOS, amd64 or arm64. Verify its entry in SHA256SUMS, make it executable, and place it on PATH.

Alternatively, with Go 1.27.4 or newer:

go install github.com/akkaraponph/evaris/cmd/evaris@v0.2.0
evaris --version

Verification

Release commit: e12566ba22fa2d63fe3ecb68ed61bf3dbd38b065. Hosted Go checks cover build, vet, race tests, offline acceptance and pinned real-Invaris integration on Linux and macOS. Binaries were built with Go 1.27.4 using the reproducible build in docs/release.md; macOS and an offline Linux container produced byte-identical binaries. Linux/arm64 and macOS/arm64 binaries passed version and experiment-validation smoke checks.

Known limitations

  • The pilot has 3 trials, one host and model, two small tasks and a stand-in instruction file; it is descriptive only and no comparison was possible. Host metadata is operator-attested from transcripts. No semantic review.
  • Command graders and Invaris guards require --allow-commands, execute trusted commands and are not sandboxed.
  • Windows and Invaris revisions other than the pinned revision are unsupported.
  • Stored v1alpha1 records have no cross-minor compatibility promise; evidence stores are not migrated.

See pilot runbook, support and reproducible builds and changelog.