Repository navigation
Releases
Evaris v0.5.0
Fixes found by evaluating a real NestJS and Next.js remaster with Evaris 0.4.0
and Invaris 0.5.0.
-
Fixed: an
allowedChangesorprotectedPathsentry always matches the
identical path. 0.4.0 read literal paths as patterns, so a listed Next.js
route such asapp/products/[id]/page.tsxno longer matched itself and an
unchanged experiment failed source authorization on a correct delivery. -
A failed
source-authorizationcriterion now has a catalogue message and
writesevidence/source-authorization.json, listing each unauthorized path,
how it changed and which rule it broke. Before, it failed with no message or
evidence. -
With verification,
allowedChanges: ["**"]is accepted whenprotectedPaths
also coversinvaris.json, so the deny-list form works with Invaris. Pattern
errors name the case, field and pattern instead ofPATH_UNSAFE: experiment. -
verification.dependencieslinks directories such asnode_modulesinto the
verification scratch root, so an Invaris campaign whose harnesses declare
workspace dependencies can run against the imported tree. Before, such a
campaign was refused withworkspace_prepare_failed. -
Campaign verification checks the complete frozen experiment and repeat set,
recomputes the native decision, and shows each experiment verdict in reports.
The real-binary CI matrix now includes tagged Invaris 0.5.0. -
Comparisons check verifier executable and input digests. Use
--left-preparationand--right-preparationto pair explicit trials. -
Import can derive observed Claude Code session metadata and confirmed
instruction reads from a retained stream JSON transcript.
Evaris v0.4.0
Acceptance mode, Git import, frozen reviewer files, per-criterion reports and
Invaris campaign verification, closing the gaps testers found when evaluating a
real Node project.
"mode": "acceptance"evaluates a single arm with any ID; reports carry
modeand no comparison, andcomparerefuses the store.allowedChangesentries are glob patterns (*,?,[...], whole-segment
**), and the newprotectedPathsalways fails authorization.limits.maxEntriesraises the 2000-entry snapshot bound (up to 200000), and
command-exitcriteria accept absolutedependenciessuch asnode_modules,
linked beside the scratch copy.- Import accepts
finalSourceof kindgit: Evaris exports the commit with
git archive, adds the prepared generated files and records the
deliveredRevision(evaris.run/v1alpha2). reviewerFilesfreezes reviewer scripts and hidden tests;evaluaterefuses
to grade when one changed.- Reports move to
evaris.report/v1alpha2with every criterion's status,
reason and message, andevaris show --evaluation ID [--criterion ID]
prints one evaluation with bounded command output. - Invaris support widens from
ee92f3cto v0.3.0, v0.4.0 and 0.5.x. Evaris
probesinvaris --version, records it, and refuses an untested release.
Findings accept an optionalline. - Verification can gate a case on an Invaris campaign (
"action": "campaign"
with frozenfiles). Every experiment in the report must have run a frozen
harness with its frozen digest; a harness read from the agent's tree is
EVIDENCE_CORRUPT. - Compatibility: 0.3.0 refuses the new optional fields, v1alpha2 runs and
reports. Invaris 0.5.0 reports that carry findinglineneed Evaris 0.4.0.
Bundled controls are synthetic, not real model measurements. Commands and Invaris guards are not sandboxed; trusted command grading requires --allow-commands. Command dependencies are outside the integrity check and their contents are not hashed. Campaign reports are checked for identity, decision, campaign digest and harness digests, not decoded in full. The earlier three-trial pilot produced NOT_COMPARABLE comparisons and does not establish agent or instruction effectiveness. Windows is not supported.
Evaris v0.3.0
Workspace initialization, npm distribution, clearer command and import failures,
and command criteria that tolerate build output. Records keep their v1alpha1
schema versions; the one new field is optional.
evaris initcreates a local workspace, and common commands use local defaults.- The
evaris-clinpm package distributes native Linux and macOS binaries. command-exitcriteria accept an optionalsnapshotExcludelist of relative
paths, such as.next, that the scratch manifests skip. A build that leaves
more than 2000 entries no longer turns a passing command into anERROR.- A command criterion that ends in
ERRORnow carries the specific reason
(LIMIT_EXCEEDED,PATH_UNSAFE) instead ofIO_ERROR, and amessagein
evaluation.jsonthat names the failing step, the command's exit code, the
kind of limit and a fix. Messages come from a fixed catalogue with no paths,
command text or oracle values; any other grader text is still cleared.
report.mddoes not show them yet. INVALID_CONFIGforagentClaimlists the accepted values, and
REVISION_MISMATCHon import says which identity field differs and what it
must be.docs/workflow.mddocuments both fields.- Compatibility: an oracle that uses
snapshotExclude, and a preparation frozen
from it, are refused by 0.2.0 and earlier as an unknown field. Oracles without
it, and every other record, are unchanged. Invaris support is still only the
pinned revisionee92f3c.
Bundled controls are synthetic, not real model measurements. Commands and Invaris guards are not sandboxed; trusted command grading requires --allow-commands. Invaris is optional and pinned to source ee92f3c00c1b59b2f576bad05569fb6eafb83097. The earlier three-trial pilot produced NOT_COMPARABLE comparisons and does not establish agent or instruction effectiveness; grading was reproduced by an independent AI operator, not a second person. Windows is not supported.
Evaris v0.2.0
Evaris 0.2.0 shows the evaluation workflow on real external agent sessions. The CLI, commands and stored schemas are unchanged from 0.1.0.
- Runbook for evaluating a real external agent: docs/pilot.md (validate, prepare, run externally, import observed metadata, evaluate, report, compare), with lessons for the Claude Code CLI host: permission mode, scratchpad location, sources for observed metadata and instruction-loading detection. Documented commands use one binary path.
- A preregistered process pilot, kept private with its evidence: 12 real Claude Code CLI sessions in 3 trials (2 original Go cases × baseline/specialist), with every
observedfield taken from the host transcript. One case separated the arms descriptively (baseline failed one stated rule in 2 of 3 trials). Every comparison was NOT_COMPARABLE because specialists read the instruction file through a shell command that the preregistered loading rule did not count. This is not evidence of agent or instruction effectiveness. - An independent AI operator session reproduced all 12 evaluations and 3 comparisons from the retained evidence. No second person reproduced the grading, and the summary was reviewed by an independent AI session.
Install
Download the binary for Linux or macOS, amd64 or arm64. Verify its entry in SHA256SUMS, make it executable, and place it on PATH.
Alternatively, with Go 1.27.4 or newer:
go install github.com/akkaraponph/evaris/cmd/evaris@v0.2.0
evaris --versionVerification
Release commit: e12566ba22fa2d63fe3ecb68ed61bf3dbd38b065. Hosted Go checks cover build, vet, race tests, offline acceptance and pinned real-Invaris integration on Linux and macOS. Binaries were built with Go 1.27.4 using the reproducible build in docs/release.md; macOS and an offline Linux container produced byte-identical binaries. Linux/arm64 and macOS/arm64 binaries passed version and experiment-validation smoke checks.
Known limitations
- The pilot has 3 trials, one host and model, two small tasks and a stand-in instruction file; it is descriptive only and no comparison was possible. Host metadata is operator-attested from transcripts. No semantic review.
- Command graders and Invaris guards require
--allow-commands, execute trusted commands and are not sandboxed. - Windows and Invaris revisions other than the pinned revision are unsupported.
- Stored
v1alpha1records have no cross-minor compatibility promise; evidence stores are not migrated.
See pilot runbook, support and reproducible builds and changelog.
Evaris v0.1.0
Evaris's first release provides an offline CLI for reproducible agent and skill evaluation.
- Validate experiments, prepare blind baseline/specialist workspaces, and import externally produced results with frozen provenance.
- Grade deterministic artifacts and source authorization, retain immutable evaluations, and generate JSON/Markdown reports with explicit missing evidence and pending review.
- Compare baseline/specialist arms and instruction revisions; mismatched or unobserved conditions remain NOT_COMPARABLE.
- Optionally verify through Invaris pinned to
ee92f3c00c1b59b2f576bad05569fb6eafb83097. Native ACCEPT/REJECT/REFUSE remains separate from the Evaris assessment. - Report the binary version through
evaris --versionandevarisVersionin reports.
Install
Download the binary for Linux or macOS, amd64 or arm64. Verify its entry in SHA256SUMS, make it executable, and place it on PATH.
Alternatively, with Go 1.27.4 or newer:
go install github.com/akkaraponph/evaris/cmd/evaris@v0.1.0
evaris --versionVerification
Release commit: d0062f4a796aa470a7e55de5943d9e9fb338a288.
Hosted Go checks cover build, vet, race tests, offline acceptance, and pinned real-Invaris integration on Linux and macOS. The macOS output-limit test now uses its own time budget and asserts an exact 4 MiB retained bound.
All four release binaries were rebuilt with Go 1.27.4 on macOS and in an offline Linux container and matched byte-for-byte across hosts. Linux/arm64 and macOS/arm64 binaries passed version and experiment-validation smoke checks. Installation through go install ...@v0.1.0 also returned evaris v0.1.0.
Known limitations
- All bundled runs are synthetic controls; no real agent or model performance has been measured. Agents must be run externally.
- Command graders and Invaris guards require
--allow-commands, execute trusted commands, and are not sandboxed. - Windows and Invaris revisions other than the pinned revision are unsupported.
- Stored
v1alpha1records have no cross-minor compatibility promise; evidence stores are not migrated.
See workflow, support and reproducible builds, and changelog.