Skip to content

v0.12.0 — evidence release: holdout protocol, e2e jobs, extraction corpus

Choose a tag to compare

View all tags
@zk-xyz zk-xyz released this 17 Jul 21:15
v0.12.0
8754f6d
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Evidence release: evaluation infrastructure that can no longer flatter itself. No tool-surface changes.

Unbiased evaluation

  • Dev/holdout split with pre-committed SHA-256 + category distribution (eval/HOLDOUT-COMMITMENTS.md); holdout prompts live outside the repo
  • 7 negative routing cases + false-positive invocation metric
  • Provider/model adapters (anthropic / openai / openai-compatible)

Completed jobs, not just first-tool choice

  • Deterministic end-to-end scenario gating the eval exit code: adopt → clarify (scripted from ground truth) → promote → recompile → context → check

Extraction truth

  • Release-gating labeled quality corpus (6 frozen fixtures, 76 tests; real extractor gaps labeled → #35)
  • Live ten-site run demoted to explicit non-blocking canary with baseline deltas

Evidence pipeline

  • Dogfood capture protocol + denylist-guarded private JSONL (real prompts feed the future holdout)
  • Per-tool output schemas: brand_status, brand_runtime, brand_check join brand_context

Full changelog: https://github.com/Brandcode-Studio/brandsystem-mcp/blob/main/CHANGELOG.md

Assets 2
Loading