Repository navigation
Releases
Releases · sx4im/skillcheck
v0.13.1
Added
- Support for
.cursorrulesand.cursor/rules/*.mdcfiles: evaluate Cursor rules directly as Markdown skills without renaming. --domainCLI option to override automatic domain inference, capped by Unicode code points to preserve multi-byte character and emoji boundaries.low_samplefield in result JSON and small-sample warning in CLI, result cards, and markdown reports when evaluating fewer than 5 tasks (Standard profile 3-task runs now warn by default).
Changed
- Replaced "double-blind" wording with "blind grading" across CLI help text, error messages, and documentation to accurately reflect single-blind grader evaluation.
- Removed unreproducible leaderboard ranking and corpus lift percentage claims from the README, landing page, and dashboard, replacing rankings with clearly labeled example reports (illustrative output, not real runs).
- Refined methodology documentation to clarify that false-verdict rates exceed ~11% when fewer than 5 tasks are evaluated and effects vary moderately or more (sd 0.30 or higher).
Fixed
- Prevented automatic domain inference from using sentences from headingless skill body text as the domain.
- Escaped Markdown formatting and user mentions (
@,[,],<,>, backticks, pipes, and newlines) in skill names and task excerpts in Markdown reports to prevent PR mention pings and table breakout. - Exited with a clear error when combining
--markdownand--jsonflags instead of silently selecting JSON.
Dependencies
- Updated dotenv to 18.0.5.
v0.13.0: Developer Habit Loop, Watch Mode, Pre-Commit Hook, Task Difficulty, and GFM Reports
Highlights of v0.13.0
- Cascading Watch Mode (
skillcheck watch <path>): Real-time file system monitoring with live token count delta tracking, instant background re-evaluation on save, and keyboard controls (rto re-run,q/Ctrl+Cto exit). - Git Pre-Commit Regression Guard (
skillcheck hook install/run): Evaluates staged skill files (git show :0:<path>) in temporary isolation to block regressions before commits land, failing closed on unexpected evaluation errors. - Task Difficulty Tuning (
--difficulty <standard | hard | adversarial>): Calibrates generator prompts with multi-constraint edge cases and counter-intuitive boundary conditions to eliminate ceiling effects on frontier models (#68). - Smart Domain Extraction: Filters generic heading filenames (
# CLAUDE.md,# AGENTS.md,# Instructions) to infer the genuine engineering domain (#69). - CI-Ready GitHub-Flavored Markdown Reports (
--markdown): Exports GFM summary tables with escaped pipes and statistical breakdowns suitable for GitHub Actions step summaries and PR comments (#70). - Resumable Evaluations (
--resume): Incremental atomic checkpointing preserving completed trial outputs, difficulty settings, and task suites across unexpected interruptions (#82, #88). - Deterministic Transcript Hashes: Canonical job sequencing (
task -> trial -> arm) guaranteeing byte-identical reproducibility regardless of concurrent resolution order (#90). - Security Hardening: Strict allow-list parsing for local
.envfiles, preventing hostile*_BASE_URLor configuration directory redirects from compromising shell API credentials. - Dashboard Leaderboard: Interactive leaderboard showcasing top featured agentic skills with live search and filter controls.
Verification
- 39 test files · 280 tests passed (100% green)
- TypeScript strict checking clean across source and tests
- ESLint clean with zero warnings
v0.11.0
Added
- Support standard
--option=valuearguments alongside space-separated options across all commands (e.g.--tasks=5,--trials=3,--output=result.json, and preserving equals signs in values/paths). (#53) - Zero-dependency cross-platform build script (
scripts/build.mjs) replacing Unix-specificrm -rf/chmodshell pipelines, enabling nativenpm run buildon Windows default shells (cmd.exe/ PowerShell). (#54) - Windows CI smoke test job validating build and binary execution on
windows-latest. (#54)
Changed
- Consolidated NVIDIA NIM provider execution into
OpenAiCompatClientwithsendChatTemplateKwargs, eliminating the single-implementationNvidiaNimClientwrapper and redundantNvidiaConfigtypes. (#52) - Typed result card formatting directly against the
EvalResultcontract, removing defensiveasRecordobject guards and loose runtime fallbacks. (#55) - Tightened landing page hero heading to "Does your skill actually help?" for improved punchiness and cleaner 2-line layout. (#56)