Skip to content

Releases: donvito/local-evals

Release list

Local Evals v0.1.0

Local Evals v0.1.0 Latest
Latest

Choose a tag to compare

View all tags
@donvito donvito released this 07 Oct 15:27
v0.1.0
9478d67
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

First source release of Local Evals, a local-first app for evaluating document → JSON, text → JSON, and tool-call proposals against OpenAI-compatible providers or OpenRouter. Inputs, outputs, scores, and execution logs are shown side by side, and all data stays in .localevals/ on your machine.

Highlights

  • Three evaluation types: Document → JSON (vision OCR + extraction), Text → JSON, and tool calling (calls are graded as proposals, never executed).
  • Guided setup in six steps, or a full form, with field rules (exact, normalized, number with tolerance, date, ignore), an optional judge model with example rubrics, and inference-only runs for unlabeled data.
  • Datasets: import JSONL/JSON manifests or dataset ZIP files (manifest + images; upload, drag-and-drop, or CLI), quick samples, provider-generated datasets, and a downloadable example ZIP with README.md and AGENTS.md for coding agents.
  • Providers: OpenRouter, OpenAI, LM Studio, Ollama, llama.cpp, or any OpenAI-compatible URL. API keys are saved once per provider and shared by its models, encrypted locally; per-model keys remain available as overrides.
  • Runs, comparison, and experiments: live progress, per-case inspection, run-to-run regression comparison, JSON/Markdown export, and named experiments.
  • CLI (npm run localevals -- --help): import, run with thresholds for CI, inspect, compare, export, and backup/restore.
  • Backups: portable backup and restore of app data, with automatic pre-upgrade copies when the database schema changes.

Getting started

Requires Node.js 22+.

npm install
npm run dev

Open http://127.0.0.1:4173. For an offline demo with synthetic data and mock responses: npm run demo, then open http://127.0.0.1:4180.

Notes

  • Databases from earlier development builds are upgraded automatically on first open (schema version 8). A copy is saved next to the database first, and existing per-model API keys move to their provider's shared key.
  • This is a source release; download the source archive below or clone the repository at tag v0.1.0.

Full changes: see the merged pull requests, including #4 (dataset ZIP import, shared provider keys, Providers/Datasets UX) and #5 (backup restore and migration fixes).

Assets 2
Loading