Skip to content

v0.1.0 - CI-backed Skill Routing Evaluation Harness

Choose a tag to compare

@Raidriar7170 Raidriar7170 released this 01 Jun 08:51

v0.1.0 - CI-backed Skill Routing Evaluation Harness

Initial public release of Hermes SkillEval.

Highlights

  • 80-task / 45-skill Hermes-style benchmark with gold and negative skill labels.
  • Keyword, hybrid, embedding, verification-gated, contrastive, and
    cross-encoder router comparisons.
  • Recall@K, MRR, NDCG, Negative Hit Rate, JSONL traces, Markdown reports, and a
    static dashboard.
  • Blind validation that caught a fine-tuned-router regression despite unchanged
    Recall@5.
  • KEEP_BASELINE release decision that preserves baseline-minilm as the
    default router.
  • CI-backed release reproducibility check with a committed manifest and artifact
    hashes.

Reviewer Entry Points

  • README: README.md
  • Release handoff: docs/release-handoff.md
  • Phase 16 blind validation comparison:
    docs/demo/phase16-blind-validation/comparison.md
  • Phase 17 release decision:
    docs/demo/phase17-calibrated-release-selector/release-decision.md
  • Phase 18 release manifest:
    docs/demo/phase18-ci-release-reproducibility/release-manifest.md

Boundary

This release presents a self-built Hermes-style skill-routing evaluation and
release-gate workflow. It does not promote the fine-tuned router as the default
and does not include model checkpoints or private machine details.