Skip to content

v3.0.1 — eval transport fix

Choose a tag to compare

@rolling-codes rolling-codes released this 02 Oct 20:11
· 125 commits to main since this release
17b6403

Patch release. Live trigger evals could not run against a current Claude Code install; this restores them.

What changed and why

eval --live fixed — native binary resolution — recent Claude Code releases ship a native bin/claude binary and no longer include the cli.js entry point that claude_process.claude_command was resolving to. On Windows the resolver rewrote the shim to node …/cli.js, which raised FileNotFoundError, got recorded as SUBPROCESS_CRASH, and surfaced as infrastructure_failed on every query. No model call was ever made, so trigger rates were silently wrong. The resolver now prefers the native binary and falls back to the legacy layout only when absent. All live paths share the same resolver: bsc.py eval --live, skill_test, run_eval, and the run_loop description optimizer. (#31)

schemaVersion moved under metadata — top-level schemaVersion is now rejected by quick_validate. This is a breaking schema change for skills written against the old layout; scripts/migrate_skill.py migrates them automatically: python -m scripts.migrate_skill /path/to/skill --to 1.

Scorer no longer rewards pre-approval grants — a skill that pre-approved all tools in its allowed-tools list was previously scored higher than one that didn't. Pre-approvals are a security-relevant decision, not a quality signal.

PRs

#31