Repository navigation
SkillEvaluator v0.8.0
SkillEvaluator 0.8.0
Tier 3 now computes every score from judge verdicts, grades accuracy on the outcome alone, and explains failing scores in the HTML report. A new tier3 feedback chat answers questions about a completed run, and opt-in policies control which models a judge may fall back to. Reports give every finding stable IDs. Tier 1 accepts audited SkillSpector baselines and evidence-bound PII waivers. This release also ships a rebuilt Tier 1 scanner image and CI bundle.
pip install --upgrade skillevaluator
skillevaluator ci initPyPI package · CLI onboarding guide · Ready-to-use CI templates
Before you upgrade
- The shared validation policy is now
default(#334). The shared OSS policy for public and private repositories was calledexternal.--profile external,--external, andSKILLEVALUATOR_PROFILE=externalstill work, with the same rules. Report policy metadata and the policy digest change. - Tier 3 scores change for the same runs (#303, #286). Accuracy grades the outcome only:
FORMAT_FOLLOWEDreplacesSKILL_IDENTIFIED, so baseline accuracy rises and accuracy lift falls. Judges return verdicts and the verifier computes every score. Eligible OpenAI judges scoregoal_accuracywith RAGAS. - A Claude Code refusal now fails the arm (#337, #347). Tier 3 trials set
CLAUDE_CODE_DISABLE_REFUSAL_FALLBACK=1. A refused attempt is not scored and fails the arm's attempt coverage, sotier3 evaluateexits 1; the run'sresult.jsonrecords the coverage error. A run that completed before can now fail; rerun it, or run the arm with another model through--agent-model claude-code=MODEL. - Tier 1 results change (#306, #295). SkillSpector agent-threat matches in files no agent reads or runs no longer fail
security. The license check parses SPDX expressions, so malformed expressions and+or-onlyforms of blocked licenses now fail.
New
skillevaluator tier3 feedback RUN_DIRopens a terminal chat grounded in one completed run's scores, comparisons, and attempts (#341).- Opt-in
LLM_JUDGE_FALLBACK_POLICY,LLM_JUDGE_RETRY_POLICY, andLLM_JUDGE_FALLBACK_MODELSfor the Tier 3 judges. The defaults keep the current behavior (#343). - Stable
finding_idandoccurrence_idon every finding. JSON addsfinding_counts,finding_groups, andmessage_counts; Markdown adds a Finding Reconciliation table; and the Markdown and HTML summaries count findings the same way (#310). --skillspector-baseline FILEapplies an audited SkillSpector suppression baseline (#300).--pii-waivers PATHaccepts a reviewed PII value bound to its file and evidence (#307).- Measured Agent Resources splits agent time into LLM and tool seconds (#344). Rubric scores and Tier 3 insights name the model that wrote them (#308). The verifier saves every judge attempt to
judge_attempts.json(#303). - Private runtime runs keep a sanitized
plugin_runtime_report.jsonthat replays offline into every report format (#345). check_public_benchmarks.py --expected-source-*checks that each benchmark card names the expected source (#340).
Fixes
- Tier 3 runs.
--stop-on-passhonors--n-concurrent(#275). Ctrl+C stops running and queued Harbor jobs (#291). Claude Code and Codex installs retry transient failures (#277). Staging works on older glibc, and sandbox errors explain Bubblewrap failures (#267). - Tier 3 grading and reports. No false security findings for
apply_patchworkspace writes (#313) or HTTP method prose (#339). Behavior evidence flags a truncated final answer (#259). HTML reports explain failing scores and show baseline trial rows (#305). - Tier 2. Large repeated-content clusters are reviewed within the prompt budget (#338, #346). Contained directory aliases such as
.agents -> .claudeno longer abort a check (#282). JSON verdicts wrapped in prose are recovered (#297). - Tier 1. Incomplete SkillSpector scans say why, for a runtime limit (#294) or an LLM provider outage (#342). PII scans skip timestamps, digests, matrices, and vectors (#304), GUIDs (#273), and Proton profile paths (#307).
argument-hintplaceholders are no longer reported as XML tags (#273).
The release pull request, #362, lists all 33 merged pull requests in a table.
Downloads
This single product release contains the Python wheel, source distribution, CI bundle, artifact manifest, and checksums. The wheel and source distribution were built from the tagged commit 8149356e4476a1497fb334727f13f699c6fa6ba5.
The Tier 1 scanner image was rebuilt for this release from main 55d2d046 in run 42158955180, and all five smoke cases passed. The CI bundle pins workflow 55d2d046595e61cce6ba9f5fd9cc84b972cd3915 and image ghcr.io/nvidia/skillevaluator-tier1@sha256:9d5483243918937be4f2178f74cf7c622a531dee41e3176906994d52fa14950a. The report publisher revision is also 55d2d046595e61cce6ba9f5fd9cc84b972cd3915. Scanner versions are unchanged from 0.7.2. The image was built before the version bump, so the package inside it reports version 0.7.2.
What's Changed
- fix(tier3): stage over existing trees on older glibc and explain bwrap failures by @rng1995 in #267
- fix(tier3): stop running and queued Harbor jobs on Ctrl+C by @rng1995 in #291
- fix(license): parse SPDX license expressions by @rng1995 in #295
- fix(tier1): avoid false positives for argument hints and GUIDs by @rng1995 in #273
- feat(tier1): apply audited SkillSpector baselines by @rng1995 in #300
- fix(tier3): retry transient Claude Code and Codex install failures by @rng1995 in #277
- fix(security): explain incomplete SkillSpector scans that hit a runtime limit by @rng1995 in #294
- fix(tier3): flag behavior evidence shortened by the final-response section limit by @xh123637 in #259
- fix(tier3): compute judge scores from verdicts and save judge evidence by @rng1995 in #303
- fix(tier3): stop apply_patch workspace writes scoring as credential access by @rng1995 in #313
- fix(tier2): allow contained directory aliases by @rng1995 in #282
- fix(pii): stop reporting timestamps, matrices, and vectors as PII by @rng1995 in #304
- fix(tier3): honor --n-concurrent with --stop-on-pass by @rng1995 in #275
- fix(tier2): review large repeated-content clusters within the prompt budget by @rng1995 in #338
- fix(tier3): score goal_accuracy with RAGAS for eligible OpenAI judges by @rng1995 in #286
- fix(tier1): scope SkillSpector agent-threat findings to agent control paths by @rng1995 in #306
- feat(reporting): reconcile stable finding IDs across report formats by @rng1995 in #310
- feat(llm): record the model that wrote rubric scores and Tier 3 insights by @rng1995 in #308
- fix(inference): recover one JSON object wrapped in LLM prose by @rng1995 in #297
- fix(ci): use a neutral default validation policy by @chrisknvidia in #334
- fix(tier3): turn off Claude Code's refusal model fallback in eval trials by @rng1995 in #337
- fix(tier3): explain failing scores and show baseline trial rows by @rng1995 in #305
- feat(tier3): add
tier3 feedbackchat over a completed run by @rng1995 in #341 - fix(security): explain SkillSpector LLM provider outages as retriable incomplete scans by @rng1995 in #342
- feat(pii): add evidence-bound PII waivers and skip Proton profile paths by @rng1995 in #307
- feat(ci): check benchmark cards against the expected source identity by @rng1995 in #340
- docs(tier3): say that a refused attempt fails the arm's attempt coverage by @rng1995 in #347
- fix(tier2): share a repeated text only when that shortens the review prompt by @rng1995 in #346
- feat(tier3): add opt-in judge fallback and strict retry policies by @rng1995 in #343
- fix(tier3): stop reading capitalized HTTP prose as an HTTPie upload by @rng1995 in #339
- Complete private runtime reports and saved-run acceptance by @rng1995 in #345
- test(tier3): start bridge worker-count tests from an idle server by @rng1995 in #358
- feat(tier3): split measured agent time into LLM and tool seconds by @rng1995 in #344
- chore(release): prepare v0.8.0 by @rng1995 in #362