v0.1.0 - CI-backed Skill Routing Evaluation Harness
v0.1.0 - CI-backed Skill Routing Evaluation Harness
Initial public release of Hermes SkillEval.
Highlights
- 80-task / 45-skill Hermes-style benchmark with gold and negative skill labels.
- Keyword, hybrid, embedding, verification-gated, contrastive, and
cross-encoder router comparisons. - Recall@K, MRR, NDCG, Negative Hit Rate, JSONL traces, Markdown reports, and a
static dashboard. - Blind validation that caught a fine-tuned-router regression despite unchanged
Recall@5. KEEP_BASELINErelease decision that preservesbaseline-minilmas the
default router.- CI-backed release reproducibility check with a committed manifest and artifact
hashes.
Reviewer Entry Points
- README:
README.md - Release handoff:
docs/release-handoff.md - Phase 16 blind validation comparison:
docs/demo/phase16-blind-validation/comparison.md - Phase 17 release decision:
docs/demo/phase17-calibrated-release-selector/release-decision.md - Phase 18 release manifest:
docs/demo/phase18-ci-release-reproducibility/release-manifest.md
Boundary
This release presents a self-built Hermes-style skill-routing evaluation and
release-gate workflow. It does not promote the fine-tuned router as the default
and does not include model checkpoints or private machine details.