Skip to content

v1.1.3 — evals: the standard tier measured on Sonnet 5.5; the plain agent is preferred

Choose a tag to compare

@github-actions github-actions released this 25 Oct 18:54
· 15 commits to main since this release
  • The standard tier now resolves to Sonnet 5.5 in Claude Code, the same generation as light (Haiku 5.5) and heavy (Opus 5.5), at the same list price as Sonnet 5. The model-router agent and the routing table's /model advice pick it up for every skill assigned standard. On the 224 held-out evals Sonnet 5 had met fewer checks than Haiku 5.5, with a skill (78% against 83%) and without one (72% against 80%); Sonnet 5.5 meets 85% and 83%, and costs $0.07 a run with a skill against Sonnet 5's $0.13.
  • The benchmark on Sonnet 5.5 is published, and it does not favour the skills. On all 308 held-out evals, two blind judges preferred the skill's result in 40% and 43% of pairs; on the 176 that no revision or admission was based on, 32% and 35%. On the older Sonnet 5 those 176 were at even, so the stronger the plain agent, the less the skill is preferred. The account, with all three tiers side by side, is in What the benchmark showed; the summary is docs/benchmarks/standard-sonnet-5-5.json. No skill's tier changed.
  • The held-out number now says what is in it. What the benchmark showed splits the 308 held-out evals into those a skill was revised against afterwards, those a new skill was admitted on, and those nothing was decided on; the home page, the evals page and the README point to it, and the home page names the model the published run used.

npx skilldrop-cli@1.1.3 install --pack solution-architect

skilldrop-cli@1.1.3 on npm, published with provenance.
The attached skilldrop-cli-1.1.3.tgz is that same file. Check its signed build provenance with
gh attestation verify skilldrop-cli-1.1.3.tgz --repo sananthanarayan/skilldrop; its SBOM is skilldrop-cli-1.1.3.cdx.json.