Skip to content

EvalGlass v0.1.0

Choose a tag to compare

View all tags
@lmeirosu lmeirosu released this 25 Jul 09:18
v0.1.0
7dcf1ff
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

EvalGlass v0.1.0 — first pre-alpha

Evaluation discipline for agentic apps — without becoming an eval team. A small, vendored,
local-first evaluation framework whose central rule is no false confidence: a green or
non-failing result never implies more evidence, authority, calibration, comparability, or safety
than the run actually earned.

Architecture

  • Effect-free Evaluation Core (standard-library-only): contracts, score status/validity,
    aggregation, provenance, structured baseline comparability, and a single Verdict Engine that
    is the only path to pass/fail/blocked/informational.
  • Runtime Harness behind six ports (dataset, trace, task-runner, judge, result-store,
    score-sink) — owns all effects, never meaning.
  • Integration-time Skill / installer — vendors the managed runtime under evals/_evalglass/,
    scaffolds host-owned assets (all proposed/uncalibrated), and writes a manifest + lock. The
    vendored runtime keeps working after the plugin and agent are removed.

Highlights

  • Primary machine artifacts (scorecard.json, runrecord.json) with typed status, validity,
    diagnostics, provenance, and diagnostic clusters; Markdown + HTML reports rendered from them.
  • Typed authority: metrics gate only after the host validates gold, approves a threshold, and
    calibrates judges — worst-source authority prevents gating past an unvalidated data source.
  • Confidence-interval–aware gating (Wilson / Student-t), structured baseline comparability, and a
    continuous drift watcher that never calls a non-comparable run a regression.
  • JSONL dataset + trace routes, OTel/OpenInference trace import, subprocess replay, fake + host
    command judges, and CI exit mapping derived only from the core verdict.
  • Metric-discovery capability and host-evaluator authoring; optional, deletable integration lanes
    (Langfuse / Phoenix / LangSmith connectors, OpenAI-compatible judge, score-sink exporters).
  • Runs on Claude Code and Codex from one canonical source.

Pre-alpha: the public API and plugin surface may still change. Apache-2.0.
See CHANGELOG.md for the full list.

Assets 2
Loading