EvalGlass v0.2.0

EvalGlass v0.2.0

The first framework release since the 0.1.0 packaging cut: the epistemic core (M7) — Estimates
with confidence intervals, decision policies, capability-typed authority, digest-bound grants,
computed judge agreement — plus metric discovery (schema/prompt mining → a typed
MetricCatalog), live trace connectors (Langfuse / Phoenix / LangSmith, opt-in), an HTML
Scorecard, config-reachable trajectory/session units, diagnostic clusters, a continuous
drift watcher, and per-metric example selectors. Every new surface holds the no-false-confidence
line: new signal stays informational/proposed/uncalibrated until the host confers authority.

Added

• Continuous drift watcher (teta; ADR 0048) — evalglass watch
  makes drift detection continuous: it runs one evaluation, compares it to the configured
  baseline, records a typed drift.json, and exits (scheduled re-invocation via cron/CI, not a
  resident daemon). It is the first production consumer of the paired-comparison core: a regression
  is flagged only when the runs are comparable and the paired interval clears zero (reusing
  the interval-licensing rule); a delta inside the interval is within_noise;
  not_comparable/missing_baseline is reported as exactly that, never "no regression". Drift is
  explanatory evidence — a typed artifact + an INFO/WARNING Diagnostic appended after the verdict —
  so it adds no exit class (the exit still derives only from the run's verdict) and the watcher
  never promotes the baseline. A "no comparable regression" result never implies quality is fine.
• Diagnostic clusters in the Scorecard (teta; ADR 0047) — a
  run's failing/non-scored items are now grouped by their shared Diagnostic.code into a typed,
  per-metric cluster view on the Scorecard, turning a flat metric ("faithfulness = 0.82") into an
  actionable failure mode ("the 18% that failed are all missing-citation cases"). Computed
  effect-free from the scores by a pure, order-invariant cluster(); carried as an additive
  Scorecard.clusters field (pre-P3 scorecards byte-identical) that _verify_consistency recomputes
  from the persisted scores, so a fabricated or hand-edited cluster fails closed. report.md and
  report.html render cluster sub-rows (<code>: N cases) from the typed field only. A non-scored
  item is grouped by cause, never coerced to 0.0; a cluster adds explanatory structure only and
  never changes a verdict or manufactures authority. This groups failure instances by cause — a
  different axis from the explorer's call-identity grouping.
• connect --live <platform> verb (teta; ADR 0046) — the
  shipped, opt-in Langfuse / Phoenix / LangSmith connector lanes are now reachable by a verb
  instead of hand-written lanes: YAML. evalglass connect --live langfuse|phoenix|langsmith scaffolds/enables
  the matching opt-in, experimental, deletable TRACE_SOURCE lane in evalglass.yaml with
  credentials as env-var NAMES
  (never literal secrets — a literal is rejected and never echoed), a fail-closed data_policy
  (defaults to unknown: egress refused until you set permitted), and enabled: true so the
  following run pulls the traces. The verb imports no provider SDK and no adapters/trace_*
  module; a missing extra/credential is a clean SKIPPED; a live pull is proposed data and
  cannot gate. Scaffold-then-run; idempotent; host config preserved. Real but opt-in, off the
  required path — never production live tracing, and never a proven pass.
• Config-reachable trajectory / session units (teta;
  ADR 0045) — a traces: route may now declare
  unit: trajectory (or session/step) to grade the whole agent run instead of one LLM
  call at a time. run_config wires the existing select_units selector and the
  trajectory_shape@1 aggregate built-in (now registered in the harness evaluator loader); an
  aggregate's egress is the worst of its members (one forbidden member blocks it,
  fail-closed). Absent unit: is byte-identical to before (defaults to call). Aggregates run
  over proposed trace data, so they stay informational and cannot gate; a trajectory with no
  member output is non_evaluable, never a fabricated 0.0. Grade the trajectory; an aggregate
  green is still not proof the agent succeeded.
• Host command judge (teta; ADR 0042) —
  adapters/judge_subprocess.py (SubprocessJudgeModel) runs a real host judge inside
  evalglass run over a JSON contract (stdin {example_id, metric, input, output, reference, rubric} → stdout {value|score, rationale}), the beta-proven subprocess pattern (like the M2
  task runner; shell=False). judge: {adapter: command, command: [...]} selects it; the fake
  adapter stays the default; egress is enforced upstream; every failure edge is non-OK evidence
  with no value. The host judge lives in evals/judges/<name>.py. Uncalibrated → informational.
• Dashboard HTML Scorecard report (teta; ADR 0043) —
  harness/report_html.py renders a self-contained report.html alongside report.md: verdict
  hero, KPI tiles, per-workflow rows with the Wilson/Student-t interval band (the epistemic
  honesty made visual), tier/authority chips, a "what this run does not claim" panel, and a
  delta vs the previous run. Theme-aware; the page adds no authority.
• Discovery scaffolder conforms to the canonical host layout (teta;
  ADR 0041) — scaffold_catalog now
  writes the canonical evals/evalglass.yaml (never a parallel *.discovered.yaml), scaffolds
  evaluators that import the vendored _evalglass runtime, and can wire an opt-in trace-
  connector route (trace_connector=). A discovered suite drops straight into an installed host.
• Metric Discovery capability (teta; ADR 0039) — the
  discovering-metrics skill drives an agentic deep read of an app's traces, prompts, code, and
  domain to derive domain-meaningful metrics anchored to operational failure modes, across a
  quality taxonomy and three tiers (runtime / reference / judge), emitting a typed MetricCatalog
  (src/evalglass/discovery/). scaffold_catalog turns that catalog into a runnable host eval
  suite — catalog JSON + METRICS_CATALOG.md, a ClaimSpec per metric, a rubric per judge metric,
  scoped evaluator stubs, and a config — every asset proposed/informational/uncalibrated:
  discovery drafts, the host decides. Validated end-to-end on a host traced with the opt-in
  Langfuse connector.
• Generic OpenAI-compatible judge lane (teta; ADR 0040)
  — adapters/judge_openai.py (OpenAICompatibleJudgeModel), the tenth optional lane behind the
  JudgeModel port: generic transport only (stdlib urllib, https-only egress, NaN-rejecting
  parse, no provider SDK), with per-metric rubrics injected by the host. Opt-in, deletable,
  authority-free — an uncalibrated judge stays informational until a host runs an agreement study.
  Complements the existing opt-in host-endpoint live-judge lane (ADR 0016).
• Epistemic Core (teta milestone M7; ADR 0038) —
  Wilson/Student-t Estimates, DecisionPolicy (lower-confidence-bound gating, min_n_effective,
  max_missing_fraction), capability-typed authority (a fake judge can never gate), digest-bound
  AuthorityGrant, computed JudgeAgreementStudy, content-addressed JudgeInstrument, optional
  ClaimSpec, load-time recomputation + atomic persistence, and paired baseline deltas. All
  additive (existing JSON still parses); one Verdict Engine.
• Authoring tier + advanced extensions (v1.1; ADR 0025)
  — /evalglass add-metric, add-judge, calibrate plus the authoring-a-metric,
  writing-a-host-evaluator, calibrating-a-judge, and promoting-a-gate skills. Every scaffolded
  asset stays proposed/uncalibrated/empty-authority; gate activation is a host YAML edit with no
  verb. connect --live is an opt-in, deletable optional lane (no provider SDK shipped);
  connect --synth is governance-only (no generator); per-source-function views remain an unbuilt
  advanced extension. Optional-lane absence never breaks the required local workflow.
• Score subject identity + /evalglass view --by-call (framework slice F1;
  ADR 0024) — Score now carries optional example_id/
  unit_id (additive provenance; no change to score meaning or the Verdict Engine), the engine
  stamps it, and an artifact-shape gate proves real runrecord.json output carries it. view
  gains --by-call, grouping scores by explicit subject identity (never by list order).
  Score→source-function mapping remains an advanced extension.
• Codex second runtime (P3; ADR 0023) — packaged
  from the same repo and the same canonical skills/ tree, no forked guidance:
  .codex-plugin/plugin.json (shared name/version/license + a Codex interface{} block),
  AGENTS.md as the Codex entry, .version-bump.json + scripts/bump-version.sh
  (--check/--audit/--set across every version surface), and a deterministic
  scripts/sync-to-codex-plugin.sh (--stage/--dry-run/--fork). Runtime independence is proven
  across runtimes (the vendored runtime is identical whichever runtime installed it). A public
  Codex marketplace listing remains a maintainer step.

Changed

• The framework is project-agnostic. Scrubbed stray customer/domain-specific names from
  framework source (three core docstrings, the discovering-metrics skill, and the discovery
  catalog) and genericized domain-specific examples. Once EvalGlass is installed in a host, all
  customization (discovered metrics, rubrics, evaluators, the judge's domain framing) lives in that
  host clone — never in the framework.

Fixed

• Vendored optional lanes were broken in a real install. The installer's namespace rewrite
  only touched import/from evalglass… statements, so an ExtensionLane's
  module="evalglass.adapters…" string literal stayed un-vendored — a vendored host's lane
  registry imported the absent evalglass namespace, breaking every opt-in lane. vendor.py
  now also rewrites pure quoted dotted module-path literals (prose/docstrings untouched). Surfaced
  by running a config-driven eval through an opt-in trace-connector lane in a vendored host.
• Trace connectors dropped the workflow name. The opt-in Phoenix and LangSmith connectors
  did not carry trace_name, so a host evaluator dispatching by workflow read every metric as
  non-applicable; they now derive it from the provider's span/run name (as the opt-in Langfuse
  connector already did). example_from_trace also now propagates the envelope's metadata onto the
  Example.
• OpenInference JSON values are parsed. _span_mapping.behavior_from_attributes parses a
  JSON-mime input/output value (a string) into the object it represents, so a connector no
  longer hands an evaluator a string it reads as non-evaluable; already-structured or plain-text
  values are left as-is.
• Phoenix connector API drift. The arize-phoenix-client kwarg is project_identifier (not
  project_name), and limit=None crashed the SDK — the opt-in Phoenix lane now passes
  limit only when the lane declares one (found by a live run against the opt-in connector).
• The installer's namespace rewrite corrupted a config filename. The module-path string
  rewrite matched any quoted "evalglass.<name>" token, so the connect --config default
  "evalglass.yaml" became "_evalglass.yaml" in the vendored runtime only — the installed
  connect --live verb then wrote a stray, orphaned _evalglass.yaml instead of the host's real
  evals/evalglass.yaml (a silent misfire: exit 0, nothing wired). The rewrite is now restricted to
  managed-package module paths (core/harness/adapters), leaving filename literals untouched.
  Invisible to the source suite — only the vendored copy was wrong; surfaced by a field test of
  connect --live in a vendored host.
• python -m …harness.cli emitted a runpy warning on every run. The harness package eagerly
  imported cli, so runpy's double-import RuntimeWarning printed on every documented invocation
  (quickstart, run, CI). cli is now exposed lazily (PEP 562) — the warning is gone for all
  entrypoints, with no change to any documented command.
• Discovery dropped its own call-site classification. The scanner computed each site's kind
  (sdk / structured_output / model_construction / invoke) only to filter it away; the
  serialized call site now carries kind so a reviewer/scaffolder can act on it.

---

Full changelog: https://github.com/Syntelesis-Lab/evalglass/blob/v0.2.0/CHANGELOG.md