OpenEval v0.1.4
OpenEval v0.1.4
OpenEval v0.1.4 is the broad stability and evaluation-flow release. It brings the Evaluate pages into one coherent workflow, makes benchmark outputs easier to inspect and compare, preserves the depth of the local observation system, and makes evidence boundaries more difficult to misread.
Release theme
The product now has a clearer progression:
- Discover a bounded suite.
- Configure a harness, model, runner, samples, parallelism, and cases.
- Watch the run with honest lifecycle and telemetry state.
- Inspect case evidence and artifacts.
- Compare outcomes and visual outputs.
- Audit accuracy and proof coverage.
- Return to Live, Collection, and Timeline to understand what changed over time.
This is product-flow coverage and release hardening. It is not a claim that every provider, visual surface, or human-review path has been freshly exercised.
Evaluate workflow
Shared information architecture
Runs, Leaderboard, Compare, Cases, New Run, and Accuracy now share one Evaluate navigation model. The ordering is consistent across the sidebar, page-level workflow strips, run detail, and narrow layouts. Active-page semantics exclude special run tools from incorrectly lighting up the parent Runs item.
Run watching and diagnostics
Run detail now presents a compact evaluation pulse with:
- current and next case;
- queued, generating, grading, result, skipped, and terminal lifecycle phases;
- progress and elapsed timing;
- run-level pass rate and resolved-case counts;
- connection and SSE state;
- bounded recent activity replay;
- All, Tools, and Results filters;
- direct navigation to case evidence.
The benchmark view now uses explicit titles and units for token/cost and throughput charts, numeric ticks rather than unlabeled gridlines, ranked bars, status legends, and accessible point descriptions. Missing token or cost instrumentation is not plotted as a false zero.
Visual and creative evaluation
The visual lane now covers multiple media and task shapes instead of treating SVG as the only creative output:
- supplied-data SVG;
- pixel-art scenes;
- CSS/HTML 3D depth;
- isometric voxel worlds;
- data-story cards;
- route-planner interfaces;
- accessible filter forms;
- dashboards, posters, tables, and responsive layouts;
- Mermaid diagrams;
- Markdown incident/runbook outputs.
Compare offers a bounded side-by-side artifact stage where identity, medium, bytes, SHA-256, viewport, and availability are visible. The UI explicitly says that artifact receipts and structural contracts do not automatically verify visual quality. Pixel, screenshot, judge, and human-review evidence remain separate tiers.
New Run and Cases
New Run now begins with a focused Core suite and offers Visual lab, Reasoning, and Everything presets. It shows planned executions and a per-case budget ceiling before launch, labels Everything as the highest-usage option, and retains explicit selection semantics.
When filters hide selected cases, the selection remains visible through a hidden-selection status row. “Clear visible” clears only the current view, “Clear all” clears the complete plan, and “Show selected” restores the selected cases into view.
Cases now gives new operators a direct Core-suite starter action and a Creative sampler path, while preserving the full case catalog and category filters.
Accuracy and evidence
The runnable corpus is now 35 cases:
- 9 agentic SWE;
- 6 reasoning;
- 6 single-tool;
- 14 visual-code.
The strict accuracy audit verifies 35/35 oracle scripts and 35/35 known-bad rejection scripts. It reports deterministic, trace, visual, LLM-judge, and manual surfaces independently:
- 192 deterministic evidence declarations;
- 4 trace declarations;
- 14 visual declarations;
- 7 LLM-judge declarations;
- manual review remains a separate, currently unclaimed tier.
The audit can therefore finish successfully while still reporting Unknown for unexercised runtime or human evidence. That distinction is intentional and is preserved in the UI, manifests, release notes, and user-flow ledger.
The product/evidence ledger now separates:
- independently runnable benchmark cases;
- capability nuclei;
- evidence-lens rows;
- 250 UX coverage stories;
- 200 proficiency-based user-flow stories.
The 200 user-flow stories are evenly distributed across brand-new, beginner, intermediate, and expert journeys. 180 are mapped to current product/source/test evidence; 20 remain explicitly pending for live-provider, assistive, or human handoff review.
Observation and transcript fidelity
Live, Collection, and Timeline remain the deepest part of OpenEval. This release preserves and hardens:
- semantic normalization for old and new Codex JSONL shapes;
- Claude and ncode project transcript parsing;
- function, custom, generic-tool, and tool-search call/result records;
- call-ID pairing where available;
- bounded arguments, results, status, and duration;
- source-qualified session identity;
- parser-version cache invalidation;
- full-history versus recent-slice population receipts;
- parseable versus detect-only inventory;
- discovered, scanned, parsed, dropped, and unscanned counts;
- measured, inferred, missing, malformed, stale, archived, and incomplete provenance;
- FTS search over bounded semantic transcript and tool evidence;
- child-trace and judge-session lineage without denominator pollution.
Raw transcript files remain authoritative and unchanged. Derived transcript windows, indexes, DOM projections, and API payloads remain bounded.
Stability, security, and recovery
This release hardens the failure modes that most often turn local evaluation work into ambiguous UI:
- run cancellation aborts in-flight work and lands terminal status in a durable order;
- terminal SSE streams close without reconnect churn;
- loading/error/retry/empty states exist on the major evaluation and observation routes;
- harness discovery distinguishes missing binaries, failed probes, wrappers, revoked OAuth, and unavailable providers;
- launch parameters are bounded and field-tagged at the API boundary;
- mutation routes enforce local or same-origin request safety;
- server-owned source inventories reject traversal and symlink escapes;
- redacted output and public-upload checks keep private paths, local databases, transcripts, and identities out of the release;
- doctor detects Node/runtime mismatch, native SQLite binding failures, stale Next builds, port occupancy, database health, and disk headroom;
- the New Run harness gate test is hermetic and does not depend on local provider authentication.
Accessibility and responsive polish
The shared shell and evaluation surfaces now include labeled dialogs, focus containment and restoration, Escape dismissal, visible close actions, live status regions, visible focus rings, touch-sized controls, native button semantics, pressed/current-page state, and screen-reader lifecycle announcements.
The primary evaluation routes were browser-checked at 390×844 with zero positive document overflow. Intentional internal scrolling remains scoped to data-heavy controls rather than leaking into the document. Theme colors use the repository’s CSS-variable-safe color mixing rules.
Release verification
The final local release gate includes:
- TypeScript typecheck;
- complete node test suite;
- ESLint with zero warnings/errors;
- optimized production build;
- strict accuracy audit;
- doctor health audit;
- public-upload audit;
git diff --check;- post-build browser smoke across Dashboard, Cases, New Run, Runs, Leaderboard, Compare, Accuracy, Live, Collection, and Timeline;
- narrow-screen checks for Cases, New Run, Compare, and Accuracy;
- no visible application-error page and no captured browser console warnings/errors.
Evidence boundary
OpenEval v0.1.4 does not claim that:
- every configured provider has fresh authenticated success;
- a byte receipt, hash, selector count, or structural contract proves pixel quality;
- source and browser checks replace full WCAG conformance or assistive-technology review;
- an LLM-judge declaration is the same thing as an executed judge verdict;
- a product-flow story is an independent agent-capability benchmark case.
Those surfaces remain visible as Unknown or Pending until the corresponding runtime, provider, assistive, or human evidence is actually attached.