Skip to content

SkillEvaluator v1.0.1

Latest

Choose a tag to compare

@chrisknvidia chrisknvidia released this 18 Nov 00:10
Immutable release. Only release title and notes can be modified.
914c15c

🎉 Introducing SkillEvaluator v1.0.1, our first complete release in the 1.0 series.

Understand how your skills load, measure how they perform, and bring automated evaluation into your workflow with HarnessCheck, NeMo Gym integration, and improved agent-assisted evaluation and repository CI.

PyPI v1.0.1 · Download CI bundle v1.0.1 · Documentation

Highlights

  • HarnessCheck: find compatibility issues that can cause your skill to be missed, rejected, or truncated across Claude Code, Codex, Copilot CLI, Hermes, OpenClaw, and OpenCode. Versioned profiles check discovery paths, frontmatter, description and content limits, invocation settings, plugin components, and conflicts with other installed skills. Findings explain what each harness does and how to fix it. Run skillevaluator harness-check ./my-skill, or see the results in Tier 1 reports. These static checks need no model key and are advisory under the default policy. HarnessCheck guide
  • NeMo Gym integration: run Tier 3 evaluations with --backend nemo-gym, using shared grading and reports. Required setup
  • More control over quality: configure token budgets, description limits, reserved-name terms, and score deductions with --quality-config. Use --profile registry for stricter author and skill-ID requirements. Quality rules · Registry policy
  • Clearer reports: refreshed HTML reports put outcomes, coverage, and next actions first. Improved budget, token, cost, and baseline reporting makes partial or skipped runs easier to interpret. Reports guide
  • Better agent-assisted evaluation: the updated SkillEvaluator skill guides your agent through installation, preparing cases from real tasks, running evaluations, and interpreting results. Install it with npx skills add NVIDIA/SkillEvaluator. Bundled example skills also gain fixes for input fixtures, script handling, and validation. Agent guide
  • More CI options: add offline pre-commit checks, export JUnit XML with -r junit, or use the reusable GitHub workflow for scored PR comments without hosted reports. Pre-commit hooks · JUnit output · Comment workflow

Add SkillEvaluator to your repository with CI integration

Install or upgrade, then run inside your repository:

python -m pip install --upgrade skillevaluator
skillevaluator ci init

Supports Python 3.12–3.14. Standard quality and security checks need no model key.

CI configuration: GitHub YAML · GitLab YAML · Docker image.

Upgrading from v0.9.0: review the author-policy, security-override, and hook-gate changes in the changelog. Existing CI workflow and image pins must be updated separately from the Python package.

New Contributors

What's Changed

View changes and contributor credits
  • Add opt-in coverage-aware grader qualification policies by @rng1995 in #497
  • fix(tier1): gate unmodeled hooks and explain AE1 reference gaps by @rng1995 in #459
  • feat(tier3): add selectable NeMo Gym execution backend by @chrisknvidia in #225
  • Fix unchecked behavior-step qualification reporting by @rng1995 in #504
  • Add security and efficiency qualification challenge pilot by @rng1995 in #501
  • Report budget overruns and unavailable baseline pass rates by @rng1995 in #505
  • docs: define every metric formula, range, units and validation status by @rng1995 in #500
  • Report Semgrep file-level parse errors as a partial scan by @chrisknvidia in #475
  • feat(cli): show the wordmark in terminals narrower than 115 columns by @chrisknvidia in #369
  • docs(ci): say ci demo only works after the setup merges by @chrisknvidia in #477
  • test(tier1): cover rubric rejection sanitization by @chrisknvidia in #478
  • fix(tier3): withhold behavior scores built on cut final answers by @rng1995 in #503
  • Install Bandit with the base package by @chrisknvidia in #487
  • Add a three-column Get started section to the README by @chrisknvidia in #488
  • fix(reporting): give exact LLM setup fixes and count named Tier 1 checks by @chrisknvidia in #367
  • fix(tier1): keep image-only SkillSpector reports incomplete when reference evidence is unmatched by @chrisknvidia in #476
  • test(tier3): cover run-directory checks on local directory downloads by @chrisknvidia in #479
  • docs(changelog): move post-0.9.0 entries to Unreleased by @rng1995 in #506
  • fix(tier1): make author and metadata.id requirements profile-controlled by @rng1995 in #499
  • fix(reporting): stop validate from repeating its header in short terminals by @chrisknvidia in #368
  • fix(tier3): validate token case IDs and fix two security false positives by @rng1995 in #507
  • fix(tier3): decode Codex array/map wrappers and make the rubric supplement budget configurable by @rng1995 in #502
  • Clarify literal final-response format checks (partial mitigation) by @rng1995 in #509
  • fix(tier3): narrow the behavior evidence gate and correct its accounting by @rng1995 in #511
  • fix(security): update SkillSpector to latest release 2.13.4 by @rng1995 in #512
  • fix(tier1): finish the author and metadata.id review round from #499 by @rng1995 in #513
  • Explain an HTTP 404 from the default LLM model by @chrisknvidia in #514
  • fix(tier3): confine security exemptions and sanitize hidden case IDs by @rng1995 in #516
  • Recognize Node scripts in Tier 1 skill type and script checks by @chrisknvidia in #474
  • fix(tier3): match home paths in default words, pair shell quotes as the shell does, and read editor views by @rng1995 in #517
  • fix(tier3): apply baseline-timeout tolerance to the printed verdict, report notes and token evidence by @rng1995 in #523
  • feat(harness): add the advisory Tier 1 harness compatibility check by @chrisknvidia in #520
  • feat(harness): add the harness-check command and compatibility panel by @chrisknvidia in #521
  • feat(harness): flag frontmatter that harness parsers reject by @chrisknvidia in #522
  • Reuse standard scored PR comments without hosted reports by @chrisknvidia in #528
  • fix(reporting): percent-encode % in SARIF artifact URIs by @0z5a in #519
  • feat(harness): flag descriptions harnesses drop, cut or hide by @chrisknvidia in #526
  • Scope comment template permissions to the comment job by @chrisknvidia in #529
  • fix(reporting): show the evidence that decided the BENCHMARK.md verdict by @rng1995 in #524
  • fix(tier3): decode freeform Codex apply_patch literals and trust an explicit exit 0 by @rng1995 in #530
  • feat(harness): flag SKILL.md locations harnesses miss or load twice by @chrisknvidia in #527
  • fix(llm): make the Anthropic refusal fallback model configurable by @lucas-saldanha-werneck in #494
  • feat(harness): measure a skill with the skills installed beside it by @chrisknvidia in #531
  • docs(readme): add a PyPI badge linking to the published package by @chrisknvidia in #535
  • feat(harness): flag SKILL.md and files over harness delivery limits by @chrisknvidia in #532
  • feat(harness): flag placeholders other harnesses pass through literally by @chrisknvidia in #533
  • feat(harness): flag invocation fields each harness ignores by @chrisknvidia in #534
  • feat(harness): flag Hermes platforms and OpenClaw requirements by @chrisknvidia in #537
  • feat(harness): check which plugin formats each harness accepts by @chrisknvidia in #538
  • fix(install): bundle uv so a plain pip install can run the Tier 1 security check by @chrisknvidia in #539
  • fix(doctor): label scanners that SkillEvaluator provisions itself by @chrisknvidia in #540
  • fix(reports): say why the default report directory was refused and how to choose another by @chrisknvidia in #541
  • fix(notices): list Bandit with the base dependencies by @chrisknvidia in #542
  • feat(harness): check the plugin components each harness delivers by @chrisknvidia in #543
  • feat(harness): take the model setup in harness-check by @chrisknvidia in #544
  • feat(harness): flag a harness release the profiles were not verified on by @chrisknvidia in #545
  • fix(reporting): write reports for non-UTF-8 file names and keep SARIF URIs valid by @rng1995 in #536
  • fix(scanners): enforce pinned SkillSpector during automatic selection by @rng1995 in #550
  • fix(tier3): keep Codex apply_patch contents in FILE CHANGES evidence by @rng1995 in #551
  • fix(tier3): mask credentials at JSON-escaped line starts in judge evidence and keep RAGAS refusals by @rng1995 in #547
  • fix(tier3): say progress stopped when a plan line is refused, and keep the running spinner moving by @rng1995 in #548
  • fix(tier3): count token usage per case correctly and keep the terminal token notes accurate by @rng1995 in #546
  • fix(harness): drop internal test notes from finding messages by @chrisknvidia in #565
  • docs(readme): mention harness compatibility in the Tier 1 row by @chrisknvidia in #566
  • fix(tier3): keep the staged case ID and custom grader explanations in NeMo Gym results by @chrisknvidia in #572
  • docs: correct starter success-summary wording by @chrisknvidia in #574
  • fix(cli): keep success metadata readable in PowerShell by @chrisknvidia in #576
  • fix(tier3): clean up run-owned detached Gym servers by @chrisknvidia in #591
  • fix(tier1): report a nested SKILL.md once on case-insensitive filesystems by @chrisknvidia in #599
  • fix(tier3): preserve pass thresholds in incomplete runs by @chrisknvidia in #580
  • fix(tier3): make native NeMo Gym Claude Code runs reach the policy model and discover skills by @chrisknvidia in #581
  • fix(hermes): validate the selected Build credential during preflight by @chrisknvidia in #583
  • fix(codex): preserve credential-prefixed native plugin launches by @chrisknvidia in #584
  • fix(reports): distinguish skipped baselines from incomplete execution by @chrisknvidia in #575
  • fix(tier3): report final budgets and incomplete run verdicts by @chrisknvidia in #594
  • fix(tier3): reserve measured usage for in-flight attempts by @chrisknvidia in #595
  • fix(bridge): preserve NVIDIA Build token usage by @chrisknvidia in #578
  • fix(tier3): complete Gym cleanup progress after teardown by @chrisknvidia in #592
  • fix(tier3): accept declarative checks in grader validation by @chrisknvidia in #582
  • fix(tier3): keep failed judging out of skill failure advice by @chrisknvidia in #579
  • fix(tier3): preserve OpenCode bash exit status in trajectories by @chrisknvidia in #585
  • fix(tier3): scope suggestion and finding evidence to the cited case by @chrisknvidia in #587
  • docs: repair onboarding page navigation by @chrisknvidia in #573
  • fix(tier3): surface declarative-check coverage in all reports by @chrisknvidia in #586
  • fix(tier3): probe Gym policy endpoint reachability in doctor by @chrisknvidia in #593
  • fix(tier3): retain price provenance and charge coverage by @chrisknvidia in #596
  • fix(tier3): project Gym OpenCode skills into the workdir by @chrisknvidia in #589
  • fix(tier3): reject failed skill lookup activation evidence by @chrisknvidia in #590
  • fix(tier3): harden detached Gym cleanup against signals and inspection errors by @chrisknvidia in #602
  • fix(tier2): name the skill whose frontmatter stops the CI comparison catalog by @rng1995 in #549
  • fix(tier3): keep script names out of template graded text and say which SKILL.md becomes my SKILL.md by @rng1995 in #571
  • fix(tier1): keep authored eval files in the Gitleaks secret scan by @rng1995 in #552
  • fix(tier1): stop deducting quality points for a missing root version by @rng1995 in #553
  • fix(ci): stop the Tier 1 runner missing a Ctrl-C, and deflake timing-dependent tests by @chrisknvidia in #568
  • fix(tier3): stop short host flags from blocking Compose sidecar values by @chrisknvidia in #598
  • fix(tier3): load evals/config.yml strictly before the Harbor run by @rng1995 in #554
  • fix(tier3): keep case actions from weakening dataset requirements by @chrisknvidia in #588
  • fix(tier1): stop reporting link-local addresses as PII IPs by @rng1995 in #555
  • fix(tier3): pin OpenCode 1.19.5 for managed local installs by @rng1995 in #558
  • fix(tier3): mask judge and finding evidence whose quotes are escaped again by @rng1995 in #577
  • Move the nv_build default model to Nemotron 3.5 Lightning by @chrisknvidia in #605
  • Name Nemotron 3.5 Lightning as the nv_build default in the docs by @chrisknvidia in #606
  • Move SkillSpector's Anthropic and Bedrock default to Claude Opus 5.5 by @chrisknvidia in #607
  • fix(harness): flag an empty Copilot CLI description by @chrisknvidia in #612
  • fix(tier3): report budget-denied arms as not run instead of empty Harbor failures by @chrisknvidia in #616
  • docs(ci): make the jq quality gate fail closed by @chrisknvidia in #617
  • docs(tier3): point the NeMo Gym example at an OpenAI Responses endpoint by @chrisknvidia in #615
  • fix(tier3): refresh an explicit Gym policy key that shares the Vertex ADC token by @chrisknvidia in #613
  • docs(ci): pin the recipe's actions to commit SHAs by @chrisknvidia in #618
  • fix(harness): Hermes lists both same-named skills (HM-DUP-NAME) by @chrisknvidia in #619
  • docs(tier3): show how to install the pinned NeMo Gym commit by @chrisknvidia in #614
  • fix(harness): count when_to_use in Claude Code's 1,536-character cut (CC-DESC-CUT) by @chrisknvidia in #620
  • Note Claude Code skill fields whose value Claude Code ignores by @chrisknvidia in #623
  • Flag a project skill with a same-named copy in a user-level skills folder by @chrisknvidia in #622
  • Show each serious finding of a failed Tier 1 check in what to fix by @chrisknvidia in #621
  • Say SkillSpector ran when a folder run rejects its scan for an unresolved reference by @chrisknvidia in #624
  • Flag skills whose descriptions are near-identical to a sibling's by @chrisknvidia in #625
  • docs(tier2): state collection manifest byte limits by @chrisknvidia in #629
  • fix(tier2): keep catalog formatting within the byte budget by @chrisknvidia in #628
  • feat(skills): teach the skillevaluator skill to install the CLI when missing by @chrisknvidia in #570
  • docs(agent-guide): start the agent guide with the npx install and an example session by @chrisknvidia in #610
  • Pin SkillSpector and verify the Gitleaks download in the CI recipe by @chrisknvidia in #630
  • Report the missing tier3 extra from doctor on a base install by @chrisknvidia in #626
  • docs(specs): add Tier 2 requirement ledger by @chrisknvidia in #650
  • docs(specs): add reports requirement ledger by @chrisknvidia in #648
  • docs(specs): add Tier 3 scoring requirement ledger by @chrisknvidia in #647
  • docs(specs): add Tier 3 datasets requirement ledger by @chrisknvidia in #646
  • fix(tier3): keep one [verifier.env] table when staging native Gym tasks by @chrisknvidia in #639
  • docs(specs): add Tier 1 requirement ledger by @chrisknvidia in #649
  • fix(reporting): describe what the BENCHMARK Overall row compares by @chrisknvidia in #635
  • Name the missing tier3 extra when evaluate runs on a base install by @chrisknvidia in #627
  • fix(tier3): complete fully budget-denied runs without execution errors by @chrisknvidia in #658
  • docs(tier2): distinguish measured saved-catalog and in-memory scale by @chrisknvidia in #641
  • Say when to use the starter text-analyzer skill in its description by @chrisknvidia in #638
  • fix(reporting): include required case checks in the BENCHMARK verdict rule by @chrisknvidia in #644
  • fix(tier3): handle malformed mcp_servers.toml on NeMo Gym as Harbor does by @chrisknvidia in #656
  • fix(reports): align benchmark headline with completed-agent verdict by @chrisknvidia in #655
  • fix(reporting): retain fully denied run budget context in HTML by @chrisknvidia in #661
  • fix(reporting): include required case checks in the HTML verdict rule by @chrisknvidia in #651
  • fix(tier3): require a model when a custom NeMo Gym policy URL is set by @chrisknvidia in #643
  • docs(tier3): state the live status of the NeMo Gym backend by @chrisknvidia in #645
  • fix(reports): avoid completed-agent callout without complete evidence by @chrisknvidia in #660
  • fix(tier3): validate coverage against admitted attempt ordinals by @chrisknvidia in #663
  • test(tier3): model advanced ctime in the MCP source rewrite race by @chrisknvidia in #659
  • fix(tier3): redact only credentials in NeMo Gym progress and logs by @chrisknvidia in #636
  • fix(reporting): keep successful execution when Tier 3 quality fails in HTML by @chrisknvidia in #634
  • fix(tier3): name Docker address-pool exhaustion and document trial network cleanup by @rng1995 in #557
  • test(secure-fs): make same-inode read mutation metadata observable by @chrisknvidia in #652
  • docs(readme): link each get-started column to its guide and fold the uv install by @chrisknvidia in #609
  • docs(skill): explain gateway keys, gate versus score and incomplete publication by @chrisknvidia in #640
  • test(tier3): time the masking linear-time checks in CPU time, in turn by @rng1995 in #665
  • Score a missing command named after the skill as the agent's choice by @chrisknvidia in #662
  • fix(tier3): skip sourced files and longer paths in the keyword write rule by @rng1995 in #559
  • fix(cli): validate the folder of a SKILL.md or workflow-rules.mdc argument by @rng1995 in #556
  • test(secure-fs): make the directory snapshot race independent of clock ticks by @chrisknvidia in #653
  • test(ci): use the intended global config in the URL-rewrite fixture by @chrisknvidia in #642
  • docs(skill): make the skillevaluator skill ask before installing, spending or writing cases by @chrisknvidia in #601
  • fix(skill): correct consent, local-mode and eval-case issues from #570/#601 review by @rng1995 in #666
  • test(tier3): model advanced ctime in the restored-mtime summary race by @chrisknvidia in #654
  • docs(dev-guide): keep the directory-snapshot race note with the other fixture notes by @rng1995 in #667
  • fix(tier3): exclude failed arms from quality headlines by @chrisknvidia in #657
  • fix(tier3): keep per-trial activation diagnostics for failed arms by @chrisknvidia in #664
  • Restore native NVIDIA report presentation and reduce clutter by @rng1995 in #560
  • docs(ci): say the first Tier 1 run installs SkillSpector and Gitleaks by @rng1995 in #669
  • fix(reporting): correct Tier 3 failure, verdict and lift wording in reports and docs by @rng1995 in #561
  • fix(tier3): end the rich progress table's line off a terminal, and find a case's trials by its recorded case by @rng1995 in #668
  • fix(tier1): keep the hook gate failure FAIL beside an incomplete scan under a policy override by @rng1995 in #563
  • fix(tier1): keep the missing-reference line check and its warning accurate by @rng1995 in #564
  • docs(readme): say what the CI template does in the get-started CI column by @chrisknvidia in #673
  • test(reporting): import the CHANGELOG helpers without the reporting test modules by @rng1995 in #672
  • docs(tier1): say a PowerShell backtick before a space is not a quoted split by @rng1995 in #674
  • fix(report): keep Steps per Eval Case chart readable with long case IDs by @chrisknvidia in #675
  • fix(tier3): keep correctly unused negative-case advice off the skill by @chrisknvidia in #671
  • fix(tier3): retain partial arm evidence in collector JSON by @chrisknvidia in #670
  • fix(tier3): mask five more credential forms in security finding evidence by @rng1995 in #562
  • fix(tier3): count a protected path after a short option and a variable by @rng1995 in #679
  • test(tier3): allow SDK setup time in invalid-region probe test by @chrisknvidia in #683
  • ci: move the pinned uv from 0.11.27 to 0.13.8 by @chrisknvidia in #685
  • fix(validators): stop the version-label regex stalling the PII scan on long identifiers by @chrisknvidia in #684
  • fix(tier3): give the Windows process-tree kill longer than a plain reap by @chrisknvidia in #687
  • fix(tier3): record the execution backend in the Tier 3 report data by @chrisknvidia in #693
  • ci: run both HTML parser compatibility versions in one job by @chrisknvidia in #686
  • docs(tier3): state that Claude Code has run live on NeMo Gym by @chrisknvidia in #692
  • fix(tier3): show the execution backend in report.html by @chrisknvidia in #694
  • docs(readme): add GitHub stars and forks badges by @chrisknvidia in #695
  • docs(harness): add a Harness Check page that shows what the check reports by @chrisknvidia in #696
  • docs(harness): show the detailed view and how to fix each finding by @chrisknvidia in #697
  • docs(harness): list what the harness check looks at and the harnesses it covers by @chrisknvidia in #698
  • docs(harness): show the harness check on its own and inside validate by @chrisknvidia in #699
  • docs(harness): document the JSON block and when the harness check helps by @chrisknvidia in #700
  • ci: run only the documentation tests for docs, changelog and readme changes by @chrisknvidia in #701
  • docs(readme): highlight the harness check under the get-started table by @chrisknvidia in #703
  • docs(readme): centre the header and add a one-line tagline by @chrisknvidia in #705
  • ci: take macOS Tier 2 off pull requests and gate the Tier 3 platform jobs by path by @chrisknvidia in #704
  • docs(readme): say what get started offers and name the harness check in its heading by @chrisknvidia in #708
  • fix(cli): validate Tier 1 before preparing scanners by @chrisknvidia in #706
  • test(tier3): accept either cooperative cancellation result by @chrisknvidia in #707
  • feat: add pre-commit hooks for offline Tier 1 skill checks by @mvanhorn in #690
  • feat(reporting): add a JUnit XML report format by @mvanhorn in #691
  • ci: pin the repository's skill callers to the released evaluator commit by @chrisknvidia in #712
  • test(tier3): time the masking linearity check on equal work by @chrisknvidia in #717
  • docs(readme): install first, then three ways to use it, and a two-step quickstart by @chrisknvidia in #756
  • fix(lint): inspect Python scripts with a UTF-8 BOM by @pqminh27 in #677
  • fix(api-caller): encode scalar OpenAPI path values by @pqminh27 in #682
  • ci: run Windows Tier 2 tests in parallel and align native Windows action pins by @chrisknvidia in #758
  • docs: simplify installation and guide dataset-first evaluations by @chrisknvidia in #709
  • fix(task-list): preserve status words inside task descriptions by @pqminh27 in #680
  • fix(text-analyzer): leave punctuation-only tokens out of the top words by @dhananjay6561 in #738
  • fix(calculator): accept signed values and reject trailing input in conversions by @dhananjay6561 in #737
  • fix(lint): avoid recursion failures in valid Python expressions by @Afloat16 in #760
  • chore(release): prepare v1.0.0 by @chrisknvidia in #600
  • feat(tier1): start yaml setup for Efficiency, Discoverability and Correctness by @westonplatter in #611
  • fix(tier3): reject ambiguous or invalid price-file rates by @Afloat16 in #749
  • docs(release): document 1.0.0 changes and upgrade notes by @rng1995 in #769
  • fix(tier3): turn off OpenCode snapshots to stop with-skill EPIPE exits by @Elarionitis in #763
  • fix(discovery): scope exclusions below the requested scan root by @Afloat16 in #746
  • fix(create-custom-grader): ship the fixtures its eval cases ask for by @chrisknvidia in #768
  • fix(api-caller): share one URL scheme check between scripts by @chrisknvidia in #767
  • fix(release): qualify runtime fixes and prepare v1.0.1 by @chrisknvidia in #773
  • chore(release): pin the verified v1.0.1 CI bundle by @chrisknvidia in #774

Full Changelog: v0.9.0...v1.0.1