OpenEval v0.1.5

OpenEval v0.1.5

OpenEval v0.1.5 is the release-hardening pass after the v0.1.4 evaluation-flow and observation-fidelity release. It makes the local install path more explicit, keeps large evidence surfaces bounded, makes judge jobs durable and auditable, and reduces the cognitive load of the Timeline, onboarding, and mobile surfaces.

Release theme

The product now has a stronger operational loop:

1. Install against a documented Node/npm toolchain.
2. Run the doctor and the same CI/release gate locally.
3. Collect raw evidence while keeping derived views bounded.
4. Review or stream reports and artifacts without unbounded memory reads.
5. Run judge jobs with persisted selections and lease-fenced receipts.
6. Read Timeline, adoption, and mixed-method evidence without collapsing denominators.
7. Publish only after the source, build, audit, and public-upload boundaries are explicit.

Installation and release hygiene

• Node 20 and npm 10 are the supported release toolchain.
• .env.example documents the local data root, harness selection, judge configuration, host allowlist, scan budget, transcript cursor secret, and SSE controls.
• npm run verify:ci runs doctor, typecheck, lint, tests, self-test, strict accuracy, and the full candidate-scope public-upload audit.
• npm run verify:release adds the optimized production build.
• CI always installs from package-lock.json with npm ci, rather than reusing a compiled dependency tree that can hide native-module drift.
• The full npm audit is clean for both production and development dependencies.

Durable, bounded evidence

Judge jobs

Judge jobs now retain their effective source, model, effort, and prompt selection. Lease ownership is checked before and after provider work, stale workers are fenced, retry limits are explicit, and a failed persistence write is surfaced as a judge failure rather than disappearing into a successful-looking job.

Transcript, artifact, and report delivery

• JSONL readers cap individual records and disclose oversized records as truncated evidence.
• Artifact metadata is collected separately from content; JSON previews are capped at 512 KiB.
• Single byte ranges stream with 206, Content-Range, Content-Length, ETag, and Accept-Ranges headers.
• Report markdown and tarball bundles spool to bounded temporary files and stream to the client with cleanup on completion or error.
• Source revisions invalidate old outcome judgments so the Timeline does not present a stale score as current.

API recovery and polling

Collection, report, artifact, cancel, and event-stream routes now return consistent JSON/SSE errors for database, missing-resource, and poll-time failures. Visibility and live polling coalesce in-flight work and preserve the last good state while a refresh is recovering.

Product polish

• Judge setup distinguishes checking, ready, unavailable, error, and retry states.
• First-run guidance is shared between the overlay and inline Getting Started surface, preventing duplicate onboarding.
• Timeline review method, evidence denominator, mixed-method, association/causation, and adoption copy is more direct.
• Mobile navigation gives the active page more room, New Run puts the useful summary/action in a more natural order, and dense evidence sections group caveats rather than making every status compete equally.
• Support, onboarding, review, timeline, transcript, artifact, API, and responsive presentation contracts have focused regression coverage.

Verification and evidence boundary

The final local release gate passed:

• npm run verify:release;
• full npm test;
• TypeScript typecheck;
• ESLint with zero warnings/errors;
• self-test: 71 pass, 0 fail, 14 gated LLM-judge skips;
• strict accuracy: 35/35 corpus, oracle, and known-bad checks;
• public-upload audit over the release candidate;
• full npm audit with 0 vulnerabilities;
• optimized Next.js production build;
• production API smoke for pages, Collection/Timeline data, reports, bounded artifact preview/ranges, event streaming, and missing-resource envelopes;
• git diff --check.

The strict accuracy report continues to mark trace, visual, and LLM-judge evidence Unknown when no corresponding runtime input was supplied. Structural artifact receipts remain distinct from pixel-quality judgment, and deterministic fixtures remain distinct from provider-backed harness success. Those distinctions are part of the release rather than gaps hidden by a single score.

Links

• README
• Changelog
• Implementation plan
• CI workflow