Repository navigation
Solution Overview
Heval helps developers and researchers compare coding harnesses using controlled programming tasks. A comparison records the harness, model, serving vendor, tasks, attempts, environment, and verifier results. The verifier grades candidate code; an agent saying it finished does not establish success.
| User | Main actions | Access |
|---|---|---|
| Evaluator | Configure evaluations, inspect trials, edit charts, publish findings | Signs in and owns experiments and reports |
| Worker operator | Pair a Linux machine, approve profiles, supply provider credentials, maintain Docker | Controls the worker and its local execution configuration |
| Report collaborator | Review or edit a saved report | Explicit viewer/editor membership |
| Public reader | Read an intentionally published revision | Revocable public share link; no worker access |
An evaluator can also operate the worker. Only the report owner manages public links and invitations. Editing a presentation does not alter recorded trial evidence.
The browser runs a React/TypeScript application built by Vite and hosted on Vercel. WorkOS provides sign-in. Convex verifies identity and stores owned workers, experiments, run claims, normalized reports, report memberships, and presentation revisions. Its durable run records are the queue; this path has no separate Redis broker.
The connected runner requires Linux, Node.js 22+, Python 3.12+, Harbor 0.23.0, and Docker Engine with Compose. The operator pairs it to an account. A Node daemon initiates outbound HTTPS connections to claim work and send status/results. A detached supervisor invokes Harbor and enforces the run deadline. Harbor manages Docker task environments, coding harnesses, and executable verifiers.
Model requests originate from the harness on the worker. Merge Gateway is a supported provider connection and routes inference to the selected model/vendor. The website neither runs Docker nor automatically provisions evaluation machines.
Read How an evaluation works for the numbered lifecycle and a worked example. Architecture diagrams shows the same responsibilities visually.
| Location | Stored or processed data |
|---|---|
| Convex | Owned machine metadata, advertised profiles, claims, experiment status, bounded monitoring snapshots, normalized reports, memberships, revisions, export metadata |
| Linux worker | Provider credentials, approved executable profiles, task snapshots, supervisor state, raw Harbor artifacts |
| Browser | Information the signed-in user can access; no provider key or Docker socket |
| Model provider | Prompts and task content the harness sends for inference |
| Private Vercel Blob | Generated PNG/ZIP presentation exports, not a central raw evaluation archive |
The browser selects advertised profiles, not arbitrary shell commands. Profile/task digests detect stale selections and task changes. Backend ownership and membership checks enforce access. The worker is for trusted owners and reviewed tasks; Docker access is powerful and this is not an open service for arbitrary hostile workloads.
| Decision | Why | Cost or alternative |
|---|---|---|
| Hosted UI and connected worker | Evaluations outlive browser sessions and use operator-controlled compute | Installation, worker availability, and raw-artifact retention are operator responsibilities |
| Convex for state and queue | Transactions coordinate claims and reactive subscriptions update the UI | Managed-service coupling; a custom API/database/queue adds operations work |
| Harbor for execution | Reuses task, environment, harness, and verifier lifecycle | Pin Harbor and check adapter compatibility instead of building another benchmark engine |
| Detached supervisor | A daemon restart does not kill the running job | Recovery depends on durable local state; host loss is different from a browser or daemon restart |
| Shared normalized rows and chart recipes | Reports, Studio, and exports use the same values | Normalization must preserve provenance, missing metrics, and outcome distinctions |
| Explicit publication | Drafts remain private until the owner publishes a revision | Revocation prevents future access; it cannot erase copies a reader already saved |
Closing the browser does not stop a run. Reconnecting the daemon can recover supervisor state and upload its outcome. A stopped heartbeat does not prove execution stopped. Claimed work is not silently replayed on another machine; compatible queued work is the distinct movable case. Cancellation of active work waits for worker acknowledgment.
The current source supports configurable parallel trials, optional execution-error retries, per-trial CPU/memory overrides, and run deadlines. The operator still approves maximum attempts and profiles. Task/trial/profile/matrix counts have no fixed product caps in this revision; positive safe-integer and ownership/compatibility validation still applies. There are up to three paired unrevoked workers per account. Resource settings and deadlines are not dollar spending limits.
Raw audit artifacts remain on worker storage. Managed evaluation provisioning, a central raw-artifact archive, retention/restore automation, and paid scheduled sweeps remain future decisions. The direct Harbor path and older Bun fixture workbench are supporting paths, not additional schedulers in the primary hosted architecture.
Private presentation exports use a separate Convex job, Chromium in Vercel Sandbox, private Blob storage, and an authenticated download endpoint. Sandbox renders presentations; it is not the current Harbor worker.
Current evaluation flow, connected runners, live monitoring, runner protocol, experiments, report permissions, and supervisor.