v4.5.0 — 9 Hermes-Inspired Evaluation Improvements

What's New

9 improvements inspired by NousResearch/hermes-agent-self-evolution, hardening evaluation rigor, data safety, and decision quality.

P0 — High Impact

• Rubric-based evaluation — Dataset examples can now include expected_behavior rubrics. The LLM judge scores against specific criteria ("should mention null safety") instead of generic correctness. Testgen agent generates rubrics automatically.
• Constraint gates — New constraint_check.py validates proposals before merge: code growth ≤30%, entry point syntax valid, test suite passes. Candidates that fail are rejected.

P1 — Important

• Evaluator weights — Configure evaluator_weights in .evolver.json for weighted scoring (e.g., {"correctness": 0.5, "latency": 0.3}). No more flat averages masking important metrics.
• Judge feedback in proposers — LLM judge comments ("missed null safety") now surface in proposer strategy, enabling targeted mutations.
• Secret detection — New secret_filter.py detects 15+ secret patterns (API keys, tokens, PEM keys). Integrated into seed_from_traces.py and dataset_health.py.
• Holdout enforcement — Final winner comparison now uses --split held_out only, preventing overfitting to training data.

P2 — Enhancements

• Code growth tracking — LOC tracked per iteration in history. Evolution chart shows LOC column with ⚠ warning when growth exceeds 30%.
• Pareto selection — read_results.py computes Pareto front across evaluators. Reports when candidates offer genuinely different tradeoffs.

P3 — New Capability

• Session mining — New mine_sessions.py extracts eval data from Claude Code session history (~/.claude/), with keyword relevance filtering and secret detection.

New Tools (all stdlib-only)

Tool	Purpose
constraint_check.py	Validate proposals against hard constraints
secret_filter.py	Detect API keys/tokens in text
mine_sessions.py	Mine Claude Code sessions for eval data

Install

npx harness-evolver@latest

---

🤖 Generated with Claude Code