v0.3.0 — Stage 2 real Codex pilot evidence-chain release

Summary

• v0.3.0 ships the Stage 2 real Codex pilot evidence chain.
• The release includes input package qualification, privacy acceptance, Codex real-runner preflight, isolated-auth smoke/preflight, frozen 4x3x1 pilot plan, real Codex 12-run execution evidence, Stage 2 evidence-gate adapter, external data-root provenance repair, and evidence-gate closeout.

Final gate posture

• benchmark_validity_gate_status = REVIEW_REQUIRED
• router_promotion_gate_decision = KEEP_BASELINE
• blocking_failure_count = 0
• remaining caveat = live_agent.overlap_status

Raw execution facts

• 12 planned runs
• 12 Codex exec invocations
• 12 runs completed with deterministic verifier output
• raw verifier-output facts: passed=6, failed=6
• these are raw verifier facts only, not performance claims

Non-claims

• No benchmark PASS is claimed.
• No evidence-gate PASS is claimed.
• No performance improvement claim is made.
• No routed-skill-over-baseline claim is made.
• No router promotion is made.
• REVIEW_REQUIRED is not treated as PASS.
• KEEP_BASELINE is preserved.

Key docs

• docs/release-notes/v0.3.0.md
• docs/evidence-map.md
• artifacts/v0.3/release/v0.3.0-release-readiness.json
• docs/human-briefs/2026-07-08-v0.3.0-release-readiness.html
• PR #30 closeout: #30
• PR #31 release prep: #31

v0.2.1

v0.2.1

These release notes describe a patch release for the post-release onboarding
cleanup after the already published v0.2.0 release. This release packages
documentation, metadata, and reusable Action onboarding consistency only.

This patch release is:

• not Marketplace Action publication;
• not GitHub API PR comments;
• not PR annotations;
• not SaaS;
• not a runtime MCP router;
• not a public ranking table;
• not a SOTA claim;
• not automatic release publication;
• not a fine-tuned default-router promotion.

Summary

v0.2.1 makes the public repository surface consistent after v0.2.0:

• The README front door is developer-tool oriented rather than a reviewer
  evidence package.
• Package metadata and hermes_skilleval.__version__ are aligned at 0.2.1.
• Copy/paste reusable Action examples use
  Raidriar7170/hermes-skilleval@v0.2.1.
• Fresh-clone local and GitHub Action trial paths are documented for external
  maintainers.
• The future demo repo plan is documented without claiming the repository
  exists.

Router Decision

The default-router decision is unchanged from v0.2.0: baseline-minilm
remains the default router, and finetuned-embedding is not approved as
default. This patch release does not change benchmark logic, model defaults, or
runtime routing behavior.

Evidence

• v0.2.0 post-release evidence:
  records the already published v0.2.0 tag and GitHub Release facts.
• post-release onboarding cleanup Human Brief:
  summarizes the README, metadata, Action, docs, and onboarding cleanup.
• v0.2.1 patch release Human Brief:
  summarizes the patch release, publication facts, validation, and boundaries.
• Phase 18 reproducibility manifest:
  records the reproducible KEEP_BASELINE release-gate decision.
• v0.2.1 post-release evidence:
  records the verified v0.2.1 tag and GitHub Release facts.

Post-release v0.2.1 publication evidence records Published: true, tag
created: true, GitHub Release created: true, and Marketplace published:
false.

Non-goals

• No Marketplace Action publication.
• No GitHub API PR comments or PR annotations.
• No SaaS dashboard.
• No runtime MCP router.
• Not a public ranking table and not a SOTA claim.
• Not automatic merge approval and not automatic release publication.
• Not finetuned-embedding default-router promotion.

v0.2.0

v0.2.0

These release notes summarize the v0.2.0 GitHub Release package. They describe
implemented capabilities and committed review evidence only. This is not a Marketplace
Action release, not PR comment automation, not a SaaS dashboard, and not a
runtime MCP router.

Summary

v0.2.0 publishes the Hermes SkillEval release-gate evidence chain and
Reusable GitHub Action RC support evidence. The default-router decision remains
KEEP_BASELINE; baseline-minilm remains the current default router; and
finetuned-embedding is not approved as default.

Implemented Capabilities

• Release-gate evidence chain: Phase 16 blind validation, Phase 17 release
  selector, and Phase 18 reproducibility keep baseline-minilm as the default.
• Reusable GitHub Action RC: the root action.yml runs
  skilleval github-action-gate for external skill-library maintainers.
• Local external-consumer action smoke: committed RC support evidence over
  consumer-shaped skills/, benchmark/, and output paths.
• Hosted consumer action smoke: one captured hosted consumer workflow run that
  calls Raidriar7170/hermes-skilleval@main and records downloaded gate
  artifacts.
• Final approval checklist: the pre-publish reviewer-facing checklist used for
  the human GO/NO-GO decision.

Evidence

• v0.2.0 release decision:
  records the pre-publish NEEDS_REVIEW, Published: false, and
  KEEP_BASELINE review package.
• Phase 16 blind validation:
  records the blind regression evidence that blocked the fine-tuned candidate.
• Phase 17 release selector:
  records KEEP_BASELINE and keeps baseline-minilm as default.
• Phase 18 reproducibility manifest:
  records the reproducible release-check evidence.
• local external-consumer action smoke:
  records ALLOW_MERGE, Recall@5 1.0, and negative_hit_rate 0.0 for the
  local RC fixture.
• hosted consumer action smoke:
  records one hosted run with downloaded gate artifacts.
• v0.2.0 final approval checklist:
  records the pre-publish GO conditions, NO-GO items, and required human
  confirmation.

Boundaries

These notes describe implemented review evidence only. This release keeps
baseline-minilm as the default router and does not approve
finetuned-embedding as default. It is not a Marketplace Action release, not
Marketplace publication, not GitHub API PR comments, not PR annotations, not
SaaS, not a runtime MCP router, not a SOTA claim, not benchmark status, not
production readiness, and not automatic merge approval.

v0.1.0 - CI-backed Skill Routing Evaluation Harness

v0.1.0 - CI-backed Skill Routing Evaluation Harness

Initial public release of Hermes SkillEval.

Highlights

• 80-task / 45-skill Hermes-style benchmark with gold and negative skill labels.
• Keyword, hybrid, embedding, verification-gated, contrastive, and
  cross-encoder router comparisons.
• Recall@K, MRR, NDCG, Negative Hit Rate, JSONL traces, Markdown reports, and a
  static dashboard.
• Blind validation that caught a fine-tuned-router regression despite unchanged
  Recall@5.
• KEEP_BASELINE release decision that preserves baseline-minilm as the
  default router.
• CI-backed release reproducibility check with a committed manifest and artifact
  hashes.

Reviewer Entry Points

• README: README.md
• Release handoff: docs/release-handoff.md
• Phase 16 blind validation comparison:
  docs/demo/phase16-blind-validation/comparison.md
• Phase 17 release decision:
  docs/demo/phase17-calibrated-release-selector/release-decision.md
• Phase 18 release manifest:
  docs/demo/phase18-ci-release-reproducibility/release-manifest.md

Boundary

This release presents a self-built Hermes-style skill-routing evaluation and
release-gate workflow. It does not promote the fine-tuned router as the default
and does not include model checkpoints or private machine details.