Skip to content

Releases

Evaris v0.5.0

Choose a tag to compare

@akkaraponph akkaraponph released this 19 Nov 13:44
2d5556a

Fixes found by evaluating a real NestJS and Next.js remaster with Evaris 0.4.0
and Invaris 0.5.0.

  • Fixed: an allowedChanges or protectedPaths entry always matches the
    identical path. 0.4.0 read literal paths as patterns, so a listed Next.js
    route such as app/products/[id]/page.tsx no longer matched itself and an
    unchanged experiment failed source authorization on a correct delivery.

  • A failed source-authorization criterion now has a catalogue message and
    writes evidence/source-authorization.json, listing each unauthorized path,
    how it changed and which rule it broke. Before, it failed with no message or
    evidence.

  • With verification, allowedChanges: ["**"] is accepted when protectedPaths
    also covers invaris.json, so the deny-list form works with Invaris. Pattern
    errors name the case, field and pattern instead of PATH_UNSAFE: experiment.

  • verification.dependencies links directories such as node_modules into the
    verification scratch root, so an Invaris campaign whose harnesses declare
    workspace dependencies can run against the imported tree. Before, such a
    campaign was refused with workspace_prepare_failed.

  • Campaign verification checks the complete frozen experiment and repeat set,
    recomputes the native decision, and shows each experiment verdict in reports.
    The real-binary CI matrix now includes tagged Invaris 0.5.0.

  • Comparisons check verifier executable and input digests. Use
    --left-preparation and --right-preparation to pair explicit trials.

  • Import can derive observed Claude Code session metadata and confirmed
    instruction reads from a retained stream JSON transcript.

Evaris v0.4.0

Choose a tag to compare

@akkaraponph akkaraponph released this 17 Nov 06:47

Acceptance mode, Git import, frozen reviewer files, per-criterion reports and
Invaris campaign verification, closing the gaps testers found when evaluating a
real Node project.

  • "mode": "acceptance" evaluates a single arm with any ID; reports carry
    mode and no comparison, and compare refuses the store.
  • allowedChanges entries are glob patterns (*, ?, [...], whole-segment
    **), and the new protectedPaths always fails authorization.
  • limits.maxEntries raises the 2000-entry snapshot bound (up to 200000), and
    command-exit criteria accept absolute dependencies such as node_modules,
    linked beside the scratch copy.
  • Import accepts finalSource of kind git: Evaris exports the commit with
    git archive, adds the prepared generated files and records the
    deliveredRevision (evaris.run/v1alpha2).
  • reviewerFiles freezes reviewer scripts and hidden tests; evaluate refuses
    to grade when one changed.
  • Reports move to evaris.report/v1alpha2 with every criterion's status,
    reason and message, and evaris show --evaluation ID [--criterion ID]
    prints one evaluation with bounded command output.
  • Invaris support widens from ee92f3c to v0.3.0, v0.4.0 and 0.5.x. Evaris
    probes invaris --version, records it, and refuses an untested release.
    Findings accept an optional line.
  • Verification can gate a case on an Invaris campaign ("action": "campaign"
    with frozen files). Every experiment in the report must have run a frozen
    harness with its frozen digest; a harness read from the agent's tree is
    EVIDENCE_CORRUPT.
  • Compatibility: 0.3.0 refuses the new optional fields, v1alpha2 runs and
    reports. Invaris 0.5.0 reports that carry finding line need Evaris 0.4.0.

Bundled controls are synthetic, not real model measurements. Commands and Invaris guards are not sandboxed; trusted command grading requires --allow-commands. Command dependencies are outside the integrity check and their contents are not hashed. Campaign reports are checked for identity, decision, campaign digest and harness digests, not decoded in full. The earlier three-trial pilot produced NOT_COMPARABLE comparisons and does not establish agent or instruction effectiveness. Windows is not supported.

Evaris v0.3.0

Choose a tag to compare

@akkaraponph akkaraponph released this 16 Nov 07:21

Workspace initialization, npm distribution, clearer command and import failures,
and command criteria that tolerate build output. Records keep their v1alpha1
schema versions; the one new field is optional.

  • evaris init creates a local workspace, and common commands use local defaults.
  • The evaris-cli npm package distributes native Linux and macOS binaries.
  • command-exit criteria accept an optional snapshotExclude list of relative
    paths, such as .next, that the scratch manifests skip. A build that leaves
    more than 2000 entries no longer turns a passing command into an ERROR.
  • A command criterion that ends in ERROR now carries the specific reason
    (LIMIT_EXCEEDED, PATH_UNSAFE) instead of IO_ERROR, and a message in
    evaluation.json that names the failing step, the command's exit code, the
    kind of limit and a fix. Messages come from a fixed catalogue with no paths,
    command text or oracle values; any other grader text is still cleared.
    report.md does not show them yet.
  • INVALID_CONFIG for agentClaim lists the accepted values, and
    REVISION_MISMATCH on import says which identity field differs and what it
    must be. docs/workflow.md documents both fields.
  • Compatibility: an oracle that uses snapshotExclude, and a preparation frozen
    from it, are refused by 0.2.0 and earlier as an unknown field. Oracles without
    it, and every other record, are unchanged. Invaris support is still only the
    pinned revision ee92f3c.

Bundled controls are synthetic, not real model measurements. Commands and Invaris guards are not sandboxed; trusted command grading requires --allow-commands. Invaris is optional and pinned to source ee92f3c00c1b59b2f576bad05569fb6eafb83097. The earlier three-trial pilot produced NOT_COMPARABLE comparisons and does not establish agent or instruction effectiveness; grading was reproduced by an independent AI operator, not a second person. Windows is not supported.

Evaris v0.2.0

Choose a tag to compare

@akkaraponph akkaraponph released this 15 Nov 14:57

Evaris 0.2.0 shows the evaluation workflow on real external agent sessions. The CLI, commands and stored schemas are unchanged from 0.1.0.

  • Runbook for evaluating a real external agent: docs/pilot.md (validate, prepare, run externally, import observed metadata, evaluate, report, compare), with lessons for the Claude Code CLI host: permission mode, scratchpad location, sources for observed metadata and instruction-loading detection. Documented commands use one binary path.
  • A preregistered process pilot, kept private with its evidence: 12 real Claude Code CLI sessions in 3 trials (2 original Go cases × baseline/specialist), with every observed field taken from the host transcript. One case separated the arms descriptively (baseline failed one stated rule in 2 of 3 trials). Every comparison was NOT_COMPARABLE because specialists read the instruction file through a shell command that the preregistered loading rule did not count. This is not evidence of agent or instruction effectiveness.
  • An independent AI operator session reproduced all 12 evaluations and 3 comparisons from the retained evidence. No second person reproduced the grading, and the summary was reviewed by an independent AI session.

Install

Download the binary for Linux or macOS, amd64 or arm64. Verify its entry in SHA256SUMS, make it executable, and place it on PATH.

Alternatively, with Go 1.27.4 or newer:

go install github.com/akkaraponph/evaris/cmd/evaris@v0.2.0
evaris --version

Verification

Release commit: e12566ba22fa2d63fe3ecb68ed61bf3dbd38b065. Hosted Go checks cover build, vet, race tests, offline acceptance and pinned real-Invaris integration on Linux and macOS. Binaries were built with Go 1.27.4 using the reproducible build in docs/release.md; macOS and an offline Linux container produced byte-identical binaries. Linux/arm64 and macOS/arm64 binaries passed version and experiment-validation smoke checks.

Known limitations

  • The pilot has 3 trials, one host and model, two small tasks and a stand-in instruction file; it is descriptive only and no comparison was possible. Host metadata is operator-attested from transcripts. No semantic review.
  • Command graders and Invaris guards require --allow-commands, execute trusted commands and are not sandboxed.
  • Windows and Invaris revisions other than the pinned revision are unsupported.
  • Stored v1alpha1 records have no cross-minor compatibility promise; evidence stores are not migrated.

See pilot runbook, support and reproducible builds and changelog.

Evaris v0.1.0

Choose a tag to compare

@akkaraponph akkaraponph released this 15 Nov 03:12

Evaris's first release provides an offline CLI for reproducible agent and skill evaluation.

  • Validate experiments, prepare blind baseline/specialist workspaces, and import externally produced results with frozen provenance.
  • Grade deterministic artifacts and source authorization, retain immutable evaluations, and generate JSON/Markdown reports with explicit missing evidence and pending review.
  • Compare baseline/specialist arms and instruction revisions; mismatched or unobserved conditions remain NOT_COMPARABLE.
  • Optionally verify through Invaris pinned to ee92f3c00c1b59b2f576bad05569fb6eafb83097. Native ACCEPT/REJECT/REFUSE remains separate from the Evaris assessment.
  • Report the binary version through evaris --version and evarisVersion in reports.

Install

Download the binary for Linux or macOS, amd64 or arm64. Verify its entry in SHA256SUMS, make it executable, and place it on PATH.

Alternatively, with Go 1.27.4 or newer:

go install github.com/akkaraponph/evaris/cmd/evaris@v0.1.0
evaris --version

Verification

Release commit: d0062f4a796aa470a7e55de5943d9e9fb338a288.
Hosted Go checks cover build, vet, race tests, offline acceptance, and pinned real-Invaris integration on Linux and macOS. The macOS output-limit test now uses its own time budget and asserts an exact 4 MiB retained bound.

All four release binaries were rebuilt with Go 1.27.4 on macOS and in an offline Linux container and matched byte-for-byte across hosts. Linux/arm64 and macOS/arm64 binaries passed version and experiment-validation smoke checks. Installation through go install ...@v0.1.0 also returned evaris v0.1.0.

Known limitations

  • All bundled runs are synthetic controls; no real agent or model performance has been measured. Agents must be run externally.
  • Command graders and Invaris guards require --allow-commands, execute trusted commands, and are not sandboxed.
  • Windows and Invaris revisions other than the pinned revision are unsupported.
  • Stored v1alpha1 records have no cross-minor compatibility promise; evidence stores are not migrated.

See workflow, support and reproducible builds, and changelog.