Skip to content

M2 — eval harness + findings (hosted path blocked; ADR-D1 measured)

Latest
Latest

Choose a tag to compare

View all tags
@ryandmonk ryandmonk released this 03 Jul 16:54
m2
b312d50
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Milestone 2: the eval harness, its findings, and the hardening the live runs forced.

  • Eval harness (PR-10): matrix runner over the same runPipeline as the CLI/demo; per-cell metrics with the two failure classes never blended; ADR-7 repair-template variant; deterministic fake-matrix CI gate; --resume; per-run crash containment with explicit error outcomes.
  • Hardening bought with real failures: typed adapter transport errors (#15), emitter refusal as failed-gate (#16, found live by containment), Ollama timeouts raised past undici's 300s ceiling (#17).
  • Findings (docs/findings.md, evidence: docs/evidence/2026-07-03-eval-rerun/, 216 audit reports): a two-local-model study — the hosted column produced zero governance observations (all 72 runs pre-generation 400: compiled grammar too large; the hosted twin of the S0 mlx finding; matched-schema design filed as #20). Local (n=72 each, 0 end-to-end passes): the ADR-D1 projection gap is the entire gate-failure story (60/60 cross-local-model, ordinary prompts) → v0.4 requiredProps headline candidate; repair-shape reading confounded (deconfound owed); the take-2 n=1 repair did not replicate (0/6).
  • ds-mcp gained get-generation-context/validate-ui (PR-11) via this repo's zero-network core, git-pinned.
  • M3 backlog: #19 (pre-generation failed-adapter classification), #20 (matched-schema hosted design), plus the plan's filed M3 agenda.

Companion M2 tags: ds-mcp@m2, dspack@m2, dspack-emit@m2.

Assets 2
Loading