Skip to content

v3.7.5 — evaluation correctness, model behavior docs, Apache 2.0 license

Latest

Choose a tag to compare

@rolling-codes rolling-codes released this 11 Nov 05:35
· 3 commits to main since this release
0ca83c2

Patch release. Commands and flags are unchanged.

Fixed

Review gate

  • A recorded independent-review skip (activation.required: false with a reason) no longer fails scripts.validate or scripts.review_gate with review-agent-missing on a skill that does not ship the toolkit's agents/*.md files. The skip is still validated for a reason, a source binding, and staleness. A skip without a reason does not waive the agent check, and a required review still needs the agent files. Reported by @chaser940428-pixel.

Evaluation engine (five correctness fixes)

  • Behavior grading scoped to one case — grade_behavior() now accepts --behavior-case and passes only the selected case's assertions to the judge. Multi-case suites require explicit selection; single-case suites auto-select. Duplicate assertion text across different cases is now permitted.

  • Grades bound to their evidence — evidence_digest() computes a SHA-256 over assertions, output files, and optional transcript. merge_grades() rejects stale model grades whose digest does not match current evidence. Legacy grades without a digest are flagged unverified: true rather than silently accepted.

  • Blank prompts rejected before dispatch — run_eval() raises ValueError for blank/whitespace-only prompts before any infra or model call. generate_tests.py also filters blank prompts before writing generated YAML fixtures.

  • Frontmatter identity used consistently — grade_behavior() provenance and bsc eval now use the name: field from SKILL.md instead of the directory name. Baselines tied to a directory name need their skill_name field updated to the declared frontmatter name.

  • Infrastructure failure diagnostics preserved — _check_infra() returns a full QueryOutcome instead of a bare bool. Failure reports include infrastructure_error (e.g. timeout, authentication, not_triggered) and infrastructure_error_detail. run_loop propagates both fields; held_out_score is null when the held-out eval itself hits an infra failure.

Documentation

  • docs/MODEL_BEHAVIOR.md now covers all four current model tiers (Opus 5.5, Sonnet 5.5, Sonnet 5, Haiku 5.5, Fable 5.2) with a tier overview, per-behavior tables linked to Anthropic's own pages, and one-run live trigger snapshots per alias.
  • Adds a Sonnet 5.5 table built from Anthropic's own prompting guide (effort, between_tools thinking, early stopping and verification, mid-turn messages, JSON tasks, tool-call handling).
  • Corrects an earlier claim that Anthropic had published no Sonnet 5.5 guide. Records what two independent evaluators (Artificial Analysis and Vals.ai) report about Sonnet 5.5, with the conditions each states.

License

The project is now licensed under the Apache License 2.0 with an Additional Terms section (permitted under Apache 2.0 §4(d)): anyone who sells or commercially distributes the Work or any Derivative Work must credit RollingCodes as the original creator with a link to this repository. Free use, modification, and redistribution are unaffected.


Thank you, @chaser940428-pixel

I hope you will contribute to this project again. It took me a long time; I hope you get use out of it. It might look like generic instructions, but it's not. I actually did hours and hours of research for it, but I really, really appreciate your contribution. I don't have as much time as I would like to work on it anymore, so I welcome and appreciate any contributions.


Full changelog: https://github.com/rolling-codes/better-skill-creator/blob/main/CHANGELOG.md#375---2026-11-10