EvalGlass v0.2.0

EvalGlass v0.2.0

The first framework release since the 0.1.0 packaging cut: the epistemic core (M7) — Estimates
with confidence intervals, decision policies, capability-typed authority, digest-bound grants,
computed judge agreement — plus metric discovery (schema/prompt mining → a typed
MetricCatalog), live trace connectors (Langfuse / Phoenix / LangSmith, opt-in), an HTML
Scorecard, config-reachable trajectory/session units, diagnostic clusters, a continuous
drift watcher, and per-metric example selectors. Every new surface holds the no-false-confidence
line: new signal stays informational/proposed/uncalibrated until the host confers authority.

Added

• Continuous drift watcher (teta; ADR 0048) — evalglass watch
  makes drift detection continuous: it runs one evaluation, compares it to the configured
  baseline, records a typed drift.json, and exits (scheduled re-invocation via cron/CI, not a
  resident daemon). It is the first production consumer of the paired-comparison core: a regression
  is flagged only when the runs are comparable and the paired interval clears zero (reusing
  the interval-licensing rule); a delta inside the interval is within_noise;
  not_comparable/missing_baseline is reported as exactly that, never "no regression". Drift is
  explanatory evidence — a typed artifact + an INFO/WARNING Diagnostic appended after the verdict —
  so it adds no exit class (the exit still derives only from the run's verdict) and the watcher
  never promotes the baseline. A "no comparable regression" result never implies quality is fine.
• Diagnostic clusters in the Scorecard (teta; ADR 0047) — a
  run's failing/non-scored items are now grouped by their shared Diagnostic.code into a typed,
  per-metric cluster view on the Scorecard, turning a flat metric ("faithfulness = 0.82") into an
  actionable failure mode ("the 18% that failed are all missing-citation cases"). Computed
  effect-free from the scores by a pure, order-invariant cluster(); carried as an additive
  Scorecard.clusters field (pre-P3 scorecards byte-identical) that _verify_consistency recomputes
  from the persisted scores, so a fabricated or hand-edited cluster fails closed. report.md and
  report.html render cluster sub-rows (<code>: N cases) from the typed field only. A non-scored
  item is grouped by cause, never coerced to 0.0; a cluster adds explanatory structure only and
  never changes a verdict or manufactures authority. This groups failure instances by cause — a
  different axis from the explorer's call-identity grouping.
• connect --live <platform> verb (teta; ADR 0046) — the
  shipped, opt-in Langfuse / Phoenix / LangSmith connector lanes are now reachable by a verb
  instead of hand-written lanes: YAML. evalglass connect --live langfuse|phoenix|langsmith scaffolds/enables
  the matching opt-in, experimental, deletable TRACE_SOURCE lane in evalglass.yaml with
  credentials as env-var NAMES
  (never literal secrets — a literal is rejected and never echoed), a fail-closed data_policy
  (defaults to unknown: egress refused until you set permitted), and enabled: true so the
  following run pulls the traces. The verb imports no provider SDK and no adapters/trace_*
  module; a missing extra/credential is a clean SKIPPED; a live pull is proposed data and
  cannot gate. Scaffold-then-run; idempotent; host config preserved. Real but opt-in, off the
  required path — never production live tracing, and never a proven pass.
• Config-reachable trajectory / session units (teta;
  ADR 0045) — a traces: route may now declare
  unit: trajectory (or session/step) to grade the whole agent run instead of one LLM
  call at a time. run_config wires the existing select_units selector and the
  trajectory_shape@1 aggregate built-in (now registered in the harness evaluator loader); an
  aggregate's egress is the worst of its members (one forbidden member blocks it,
  fail-closed). Absent unit: is byte-identical to before (defaults to call). Aggregates run
  over proposed trace data, so they stay informational and cannot gate; a trajectory with no
  member output is non_evaluable, never a fabricated 0.0. Grade the trajectory; an aggregate
  green is still not proof the agent succeeded.
• Host command judge (teta; ADR 0042) —
  adapters/judge_subprocess.py (SubprocessJudgeModel) runs a real host judge inside
  evalglass run over a JSON contract (stdin {example_id, metric, input, output, reference, rubric} → stdout {value|score, rationale}), the beta-proven subprocess pattern (like the M2
  task runner; shell=False). judge: {adapter: command, command: [...]} selects it; the fake
  adapter stays the default; egress is enforced upstream; every failure edge is non-OK evidence
  with no value. The host judge lives in evals/judges/<name>.py. Uncalibrated → informational.
• Dashboard HTML Scorecard report (teta; ADR 0043) —
  harness/report_html.py renders a self-contained report.html alongside report.md: verdict
  hero, KPI tiles, per-workflow rows with the Wilson/Student-t interval band (the epistemic
  honesty made visual), tier/authority chips, a "what this run does not claim" panel, and a
  delta vs the previous run. Theme-aware; the page adds no authority.
• Discovery scaffolder conforms to the canonical host layout (teta;
  ADR 0041) — scaffold_catalog now
  writes the canonical evals/evalglass.yaml (never a parallel *.discovered.yaml), scaffolds
  evaluators that import the vendored _evalglass runtime, and can wire an opt-in trace-
  connector route (trace_connector=). A discovered suite drops straight into an installed host.
• Metric Discovery capability (teta; ADR 0039) — the
  discovering-metrics skill drives an agentic deep read of an app's traces, prompts, code, and
  domain to derive domain-meaningful metrics anchored to operational failure modes, across a
  quality taxonomy and three tiers (runtime / reference / judge), emitting a typed MetricCatalog
  (src/evalglass/discovery/). scaffold_catalog turns that catalog into a runnable host eval
  suite — catalog JSON + METRICS_CATALOG.md, a ClaimSpec per metric, a rubric per judge metric,
  scoped evaluator stubs, and a config — every asset proposed/informational/uncalibrated:
  discovery drafts, the host decides. Validated end-to-end on a host traced with the opt-in
  Langfuse connector.
• Generic OpenAI-compatible judge lane (teta; ADR 0040)
  — adapters/judge_openai.py (OpenAICompatibleJudgeModel), the tenth optional lane behind the
  JudgeModel port: generic transport only (stdlib urllib, https-only egress, NaN-rejecting
  parse, no provider SDK), with per-metric rubrics injected by the host. Opt-in, deletable,
  authority-free — an uncalibrated judge stays informational until a host runs an agreement study.
  Complements the existing opt-in host-endpoint live-judge lane (ADR 0016).
• Epistemic Core (teta milestone M7; ADR 0038) —
  Wilson/Student-t Estimates, DecisionPolicy (lower-confidence-bound gating, min_n_effective,
  max_missing_fraction), capability-typed authority (a fake judge can never gate), digest-bound
  AuthorityGrant, computed JudgeAgreementStudy, content-addressed JudgeInstrument, optional
  ClaimSpec, load-time recomputation + atomic persistence, and paired baseline deltas. All
  additive (existing JSON still parses); one Verdict Engine.
• Authoring tier + advanced extensions (v1.1; ADR 0025)
  — /evalglass add-metric, add-judge, calibrate plus the authoring-a-metric,
  writing-a-host-evaluator, calibrating-a-judge, and promoting-a-gate skills. Every scaffolded
  asset stays proposed/uncalibrated/empty-authority; gate activation is a host YAML edit with no
  verb. connect --live is an opt-in, deletable optional lane (no provider SDK shipped);
  connect --synth is governance-only (no generator); per-source-function views remain an unbuilt
  advanced extension. Optional-lane absence never breaks the required local workflow.
• Score subject identity + /evalglass view --by-call (framework slice F1;
  ADR 0024) — Score now carries optional example_id/
  unit_id (additive provenance; no change to score meaning or the Verdict Engine), the engine
  stamps it, and an artifact-shape gate proves real runrecord.json output carries it. view
  gains --by-call, grouping scores by explicit subject identity (never by list order).
  Score→source-function mapping remains an advanced extension.
• Codex second runtime (P3; ADR 0023) — packaged
  from the same repo and the same canonical skills/ tree, no forked guidance:
  .codex-plugin/plugin.json (shared name/version/license + a Codex interface{} block),
  AGENTS.md as the Codex entry, .version-bump.json + scripts/bump-version.sh
  (--check/--audit/--set across every version surface), and a deterministic
  scripts/sync-to-codex-plugin.sh (--stage/--dry-run/--fork). Runtime independence is proven
  across runtimes (the vendored runtime is identical whichever runtime installed it). A public
  Codex marketplace listing remains a maintainer step.

Changed

• **The framework...

EvalGlass v0.1.0

EvalGlass v0.1.0 — first pre-alpha

Evaluation discipline for agentic apps — without becoming an eval team. A small, vendored,
local-first evaluation framework whose central rule is no false confidence: a green or
non-failing result never implies more evidence, authority, calibration, comparability, or safety
than the run actually earned.

Architecture

• Effect-free Evaluation Core (standard-library-only): contracts, score status/validity,
  aggregation, provenance, structured baseline comparability, and a single Verdict Engine that
  is the only path to pass/fail/blocked/informational.
• Runtime Harness behind six ports (dataset, trace, task-runner, judge, result-store,
  score-sink) — owns all effects, never meaning.
• Integration-time Skill / installer — vendors the managed runtime under evals/_evalglass/,
  scaffolds host-owned assets (all proposed/uncalibrated), and writes a manifest + lock. The
  vendored runtime keeps working after the plugin and agent are removed.

Highlights

• Primary machine artifacts (scorecard.json, runrecord.json) with typed status, validity,
  diagnostics, provenance, and diagnostic clusters; Markdown + HTML reports rendered from them.
• Typed authority: metrics gate only after the host validates gold, approves a threshold, and
  calibrates judges — worst-source authority prevents gating past an unvalidated data source.
• Confidence-interval–aware gating (Wilson / Student-t), structured baseline comparability, and a
  continuous drift watcher that never calls a non-comparable run a regression.
• JSONL dataset + trace routes, OTel/OpenInference trace import, subprocess replay, fake + host
  command judges, and CI exit mapping derived only from the core verdict.
• Metric-discovery capability and host-evaluator authoring; optional, deletable integration lanes
  (Langfuse / Phoenix / LangSmith connectors, OpenAI-compatible judge, score-sink exporters).
• Runs on Claude Code and Codex from one canonical source.

Pre-alpha: the public API and plugin surface may still change. Apache-2.0.
See CHANGELOG.md for the full list.