Repository navigation
SkillEvaluator 0.9.0
SkillEvaluator 0.9.0 expands how you build evaluation datasets, compare skill versions, and understand why an agent passed or failed. It adds controls for evaluation budgets, reusable baselines, and clearer evidence across terminal, HTML, JSON, Markdown, and benchmark reports.
For PyPI users upgrading from 0.7.2, this release also includes the changes shipped in the GitHub-only 0.8.0 release.
Highlights
- Build and review better evaluation datasets. Declare deterministic checks for script execution, workspace files, and final responses. Generate datasets grounded in skill instructions, author intent, or traces; import production traces as redacted review candidates; and promote expectations only after explicit human review. Preview legacy dataset migrations before writing a new file.
- Track skill improvements over time. Compare retained skill versions with
--compare-toandcompare-versions, browse run history, and reuse a compatible no-skill baseline with--reuse-baseline. Version comparisons remain separate from Skill Lift against a no-skill baseline. - Control evaluation spending. Set token, judge-call, attempt, and priced-cost budgets. Reports retain partial results when limits stop a run. Token usage is shown per case and agent, with missing counters and reused-baseline usage identified.
- Understand the evidence behind each result. Reports show met, unmet, unchecked, and inapplicable expectations; dataset coverage and sufficiency; skill activation; failed-arm coverage; and concrete next steps. Skill Lift is the headline comparison, with pass@k shown as supporting reliability. HTML charts work offline.
- Audit graders and judge decisions. Qualify the shipped verifier with
tier3 validate-grader, inspect retained judge requests and replies, use opt-in judge retry and fallback policies, and ask questions about a completed run withtier3 feedback. Import externally computed custom metrics with their provenance. - Extend repository CI. Opt-in GitHub and GitLab setup adds advisory Tier 2 and Tier 3 evaluations alongside required Tier 1 checks. Capability-gated onboarding supports independent chat, judge, and agent models, reviewed scanner baselines, retained benchmark cards, and optional trusted feedback.
Notable fixes
- Align Skill Lift, dimension applicability, verdicts, and publication status across report formats. Failed baselines are labeled incomplete rather than “not run,” and missing evidence stays visible.
- Improve Tier 2 handling of large repeated-content clusters, malformed LLM verdicts, collection coverage, and shell-script chunking. Failed catalog rebuilds preserve the previously committed registry.
- Reduce security and PII false positives while retaining real coverage gaps and scanner failures. Dependency advisories now appear as structured findings in reports and SARIF; stable finding IDs support tracking across runs.
- Improve agent setup retries, cancellation, stop-on-pass concurrency, UTF-8 dataset loading, retained feedback, and Docker setup. Upgrade the Tier 3 backend to Harbor 0.25.0.
Upgrade considerations
- Dataset generation now defaults to
autogrounding. Provide substantive author guidance or usable matching traces. Skill-only starter flows need--grounding skill, including with--no-llm. Older datasets that require skill use may need outcome-only expectations to pass strict CI validation; the no-skill baseline receives the same checks. - Some grading outcomes change. Accuracy evaluates task outcomes without penalizing a no-skill baseline for not identifying the skill. Standard judge scores are computed from verdicts. A scored attempt fails when goal accuracy or security is below 1.0; a scored dimension below 0.40 fails the Tier 3 verdict even when another dimension has no score. Re-evaluate before comparing scores with older releases.
- Baseline timeout tolerance is opt-in.
--tolerate-baseline-timeoutsapplies only to narrowly qualified baseline execution timeouts. It leaves that agent’s baseline and Skill Lift unavailable and does not make the run complete publication evidence. - Declare build tools explicitly. Generated Debian-based OpenCode images no longer include
gcc,g++, ormake. Skills or tasks needing them should declare packages inscripts/apt-packages.txtor a custom evaluation Dockerfile. - All-tier CI keeps Tier 3 advisory. The deprecated
--tier3-required,tier3-required, andSKILL_EVAL_TIER3_REQUIREDsettings no longer make Tier 3 a required CI gate. Standalone validation and benchmark publication policies are unchanged. - Python catalog rebuilds replace the selection. Callers reusing a registry must supply the complete intended catalog; successful builds no longer merge into the previous selection.
- Existing CI stays pinned. A Python package upgrade does not silently change an existing repository workflow. Review image and workflow pin updates together, preserving local configuration. The starter walkthrough remains pinned to its tested artifact.
- Hermes support is experimental. The new agent supports Docker with native NVIDIA Build routing; existing agent defaults are unchanged.
Add SkillEvaluator to your repository with CI integration
python -m pip install --upgrade skillevaluator
skillevaluator ci initThe generated workflow uses the release's prebuilt Docker image.
0.9.0 CI artifacts: use these versioned files for manual setup:
- GitHub checks YAML
- GitLab checks YAML
- Optional GitHub report publisher YAML
- Optional LLM checks YAML
- Complete CI bundle ZIP
Docker image: verified 0.9.0 image on GitHub Container Registry.
ghcr.io/nvidia/skillevaluator-tier1@sha256:e96e3aaad89bf78c4f3f8c621e88005d18bd7c485b29c0d7945864700e5944e6
Python package
Published on PyPI: SkillEvaluator 0.9.0. Published through Kitmaker and verified with fresh pip and uv installations.
With Python 3.12, 3.13, or 3.14, create and activate a virtual environment, then install:
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade 'skillevaluator==0.9.0'For optional Tier 2, Tier 3, and Python security dependencies:
python -m pip install --upgrade 'skillevaluator[all]==0.9.0'The optional dashboard uses the separate dashboard extra. Full security scanning also needs scanner executables; live evaluation needs provider credentials and the selected agent/sandbox runtime. See the installation guide.
See the full changelog and documentation.