Skip to content

SkillEvaluator v0.8.0

Choose a tag to compare

@rng1995 rng1995 released this 06 Nov 19:59
Immutable release. Only release title and notes can be modified.
8149356

SkillEvaluator 0.8.0

Tier 3 now computes every score from judge verdicts, grades accuracy on the outcome alone, and explains failing scores in the HTML report. A new tier3 feedback chat answers questions about a completed run, and opt-in policies control which models a judge may fall back to. Reports give every finding stable IDs. Tier 1 accepts audited SkillSpector baselines and evidence-bound PII waivers. This release also ships a rebuilt Tier 1 scanner image and CI bundle.

pip install --upgrade skillevaluator
skillevaluator ci init

PyPI package · CLI onboarding guide · Ready-to-use CI templates

Before you upgrade

  • The shared validation policy is now default (#334). The shared OSS policy for public and private repositories was called external. --profile external, --external, and SKILLEVALUATOR_PROFILE=external still work, with the same rules. Report policy metadata and the policy digest change.
  • Tier 3 scores change for the same runs (#303, #286). Accuracy grades the outcome only: FORMAT_FOLLOWED replaces SKILL_IDENTIFIED, so baseline accuracy rises and accuracy lift falls. Judges return verdicts and the verifier computes every score. Eligible OpenAI judges score goal_accuracy with RAGAS.
  • A Claude Code refusal now fails the arm (#337, #347). Tier 3 trials set CLAUDE_CODE_DISABLE_REFUSAL_FALLBACK=1. A refused attempt is not scored and fails the arm's attempt coverage, so tier3 evaluate exits 1; the run's result.json records the coverage error. A run that completed before can now fail; rerun it, or run the arm with another model through --agent-model claude-code=MODEL.
  • Tier 1 results change (#306, #295). SkillSpector agent-threat matches in files no agent reads or runs no longer fail security. The license check parses SPDX expressions, so malformed expressions and + or -only forms of blocked licenses now fail.

New

  • skillevaluator tier3 feedback RUN_DIR opens a terminal chat grounded in one completed run's scores, comparisons, and attempts (#341).
  • Opt-in LLM_JUDGE_FALLBACK_POLICY, LLM_JUDGE_RETRY_POLICY, and LLM_JUDGE_FALLBACK_MODELS for the Tier 3 judges. The defaults keep the current behavior (#343).
  • Stable finding_id and occurrence_id on every finding. JSON adds finding_counts, finding_groups, and message_counts; Markdown adds a Finding Reconciliation table; and the Markdown and HTML summaries count findings the same way (#310).
  • --skillspector-baseline FILE applies an audited SkillSpector suppression baseline (#300). --pii-waivers PATH accepts a reviewed PII value bound to its file and evidence (#307).
  • Measured Agent Resources splits agent time into LLM and tool seconds (#344). Rubric scores and Tier 3 insights name the model that wrote them (#308). The verifier saves every judge attempt to judge_attempts.json (#303).
  • Private runtime runs keep a sanitized plugin_runtime_report.json that replays offline into every report format (#345).
  • check_public_benchmarks.py --expected-source-* checks that each benchmark card names the expected source (#340).

Fixes

  • Tier 3 runs. --stop-on-pass honors --n-concurrent (#275). Ctrl+C stops running and queued Harbor jobs (#291). Claude Code and Codex installs retry transient failures (#277). Staging works on older glibc, and sandbox errors explain Bubblewrap failures (#267).
  • Tier 3 grading and reports. No false security findings for apply_patch workspace writes (#313) or HTTP method prose (#339). Behavior evidence flags a truncated final answer (#259). HTML reports explain failing scores and show baseline trial rows (#305).
  • Tier 2. Large repeated-content clusters are reviewed within the prompt budget (#338, #346). Contained directory aliases such as .agents -> .claude no longer abort a check (#282). JSON verdicts wrapped in prose are recovered (#297).
  • Tier 1. Incomplete SkillSpector scans say why, for a runtime limit (#294) or an LLM provider outage (#342). PII scans skip timestamps, digests, matrices, and vectors (#304), GUIDs (#273), and Proton profile paths (#307). argument-hint placeholders are no longer reported as XML tags (#273).

The release pull request, #362, lists all 33 merged pull requests in a table.

Downloads

This single product release contains the Python wheel, source distribution, CI bundle, artifact manifest, and checksums. The wheel and source distribution were built from the tagged commit 8149356e4476a1497fb334727f13f699c6fa6ba5.

The Tier 1 scanner image was rebuilt for this release from main 55d2d046 in run 42158955180, and all five smoke cases passed. The CI bundle pins workflow 55d2d046595e61cce6ba9f5fd9cc84b972cd3915 and image ghcr.io/nvidia/skillevaluator-tier1@sha256:9d5483243918937be4f2178f74cf7c622a531dee41e3176906994d52fa14950a. The report publisher revision is also 55d2d046595e61cce6ba9f5fd9cc84b972cd3915. Scanner versions are unchanged from 0.7.2. The image was built before the version bump, so the package inside it reports version 0.7.2.

Full changelog

What's Changed

  • fix(tier3): stage over existing trees on older glibc and explain bwrap failures by @rng1995 in #267
  • fix(tier3): stop running and queued Harbor jobs on Ctrl+C by @rng1995 in #291
  • fix(license): parse SPDX license expressions by @rng1995 in #295
  • fix(tier1): avoid false positives for argument hints and GUIDs by @rng1995 in #273
  • feat(tier1): apply audited SkillSpector baselines by @rng1995 in #300
  • fix(tier3): retry transient Claude Code and Codex install failures by @rng1995 in #277
  • fix(security): explain incomplete SkillSpector scans that hit a runtime limit by @rng1995 in #294
  • fix(tier3): flag behavior evidence shortened by the final-response section limit by @xh123637 in #259
  • fix(tier3): compute judge scores from verdicts and save judge evidence by @rng1995 in #303
  • fix(tier3): stop apply_patch workspace writes scoring as credential access by @rng1995 in #313
  • fix(tier2): allow contained directory aliases by @rng1995 in #282
  • fix(pii): stop reporting timestamps, matrices, and vectors as PII by @rng1995 in #304
  • fix(tier3): honor --n-concurrent with --stop-on-pass by @rng1995 in #275
  • fix(tier2): review large repeated-content clusters within the prompt budget by @rng1995 in #338
  • fix(tier3): score goal_accuracy with RAGAS for eligible OpenAI judges by @rng1995 in #286
  • fix(tier1): scope SkillSpector agent-threat findings to agent control paths by @rng1995 in #306
  • feat(reporting): reconcile stable finding IDs across report formats by @rng1995 in #310
  • feat(llm): record the model that wrote rubric scores and Tier 3 insights by @rng1995 in #308
  • fix(inference): recover one JSON object wrapped in LLM prose by @rng1995 in #297
  • fix(ci): use a neutral default validation policy by @chrisknvidia in #334
  • fix(tier3): turn off Claude Code's refusal model fallback in eval trials by @rng1995 in #337
  • fix(tier3): explain failing scores and show baseline trial rows by @rng1995 in #305
  • feat(tier3): add tier3 feedback chat over a completed run by @rng1995 in #341
  • fix(security): explain SkillSpector LLM provider outages as retriable incomplete scans by @rng1995 in #342
  • feat(pii): add evidence-bound PII waivers and skip Proton profile paths by @rng1995 in #307
  • feat(ci): check benchmark cards against the expected source identity by @rng1995 in #340
  • docs(tier3): say that a refused attempt fails the arm's attempt coverage by @rng1995 in #347
  • fix(tier2): share a repeated text only when that shortens the review prompt by @rng1995 in #346
  • feat(tier3): add opt-in judge fallback and strict retry policies by @rng1995 in #343
  • fix(tier3): stop reading capitalized HTTP prose as an HTTPie upload by @rng1995 in #339
  • Complete private runtime reports and saved-run acceptance by @rng1995 in #345
  • test(tier3): start bridge worker-count tests from an idle server by @rng1995 in #358
  • feat(tier3): split measured agent time into LLM and tool seconds by @rng1995 in #344
  • chore(release): prepare v0.8.0 by @rng1995 in #362

New Contributors