SkillEvaluator v0.4.0

SkillEvaluator v0.4.0 simplifies running evaluations and strengthens reporting, security checks, and large-catalog analysis.

Highlights

• Run tiers directly with skillevaluator tier1 PATH, tier2 PATH, or tier3 PATH, with clearer provider setup, gateway model defaults, and evaluation progress.
• Scan larger Tier 2 collections with explicit entry and comparison budgets, faster vector comparisons, and actionable service-failure diagnostics.
• Improve Tier 3 script-invocation evidence, protected-path write detection, token masking, and incomplete-run feedback.
• Harden CLI rendering and secret redaction, run the Docker image as an unprivileged user, and pin the HTML report's Chart.js dependency with integrity verification.

Upgrade notes

• validate PATH now runs all three tiers for skills by default. Use --tiers 1 for keyless static CI; --tiers, --no-tier3, and --no-autopilot provide explicit scope controls.
• When --agents is omitted, Tier 3 selects a provider-native agent. Extra agent runtime preflight is opt-in through --agent-runtime-preflight.
• The NVIDIA Build defaults are now nvidia/nemotron-3-super-120b-a12b for chat and nvidia/nemotron-3-embed-1b for embeddings. Rebuild existing catalogs and caches when switching embedding models.

See the v0.4.0 changelog for the full release notes.

What's Changed

• test: stabilize FIFO subprocess checks by @deepujain in #148
• docs: link SkillEvaluator research paper by @chrisknvidia in #146
• docs: add docs/livestream/blog links to README by @mosheabr in #159
• feat(cli): simplify tier workflows and improve evaluation feedback by @chrisknvidia in #152
• fix(reporting): verify the pinned Chart.js dependency by @deepujain in #162
• fix(container): use an unprivileged runtime account by @deepujain in #163
• Keep fenced code within its Markdown section during chunking by @sylvesterkaczmarek in #165
• fix: keep --no-llm negative cases off-skill by @mimran-khan in #100
• fix: harden all CI workflow guards by @efegokdemir in #158
• fix(tier2): support larger bounded collection scans by @rng1995 in #166
• fix(tier3): require invocation evidence for script execution credit by @mavericksea-ai in #154
• fix(security): stop reporting User-Agent versions as IP addresses by @JayOfTheKeyboard in #168
• fix(reporting): escape untrusted text in Rich CLI output by @rng1995 in #173
• fix(tier3): detect apply_patch writes to protected paths and mask GitHub tokens by @rng1995 in #174
• fix(redaction): make sensitive-key, JWT, and private-key matching linear-time by @rng1995 in #172
• fix(reporting): strip terminal controls from catalog and view output by @rng1995 in #175
• fix(tier3): make log JWT redaction linear-time by @rng1995 in #176
• chore(release): prepare v0.4.0 by @rng1995 in #177

New Contributors

• @sylvesterkaczmarek made their first contribution in #165
• @efegokdemir made their first contribution in #158
• @mavericksea-ai made their first contribution in #154
• @JayOfTheKeyboard made their first contribution in #168

Full Changelog: v0.3.0...v0.4.0