Skip to content

Releases

v0.13.1

Choose a tag to compare

@sx4im sx4im released this 10 Oct 16:30
c95374b

Added

  • Support for .cursorrules and .cursor/rules/*.mdc files: evaluate Cursor rules directly as Markdown skills without renaming.
  • --domain CLI option to override automatic domain inference, capped by Unicode code points to preserve multi-byte character and emoji boundaries.
  • low_sample field in result JSON and small-sample warning in CLI, result cards, and markdown reports when evaluating fewer than 5 tasks (Standard profile 3-task runs now warn by default).

Changed

  • Replaced "double-blind" wording with "blind grading" across CLI help text, error messages, and documentation to accurately reflect single-blind grader evaluation.
  • Removed unreproducible leaderboard ranking and corpus lift percentage claims from the README, landing page, and dashboard, replacing rankings with clearly labeled example reports (illustrative output, not real runs).
  • Refined methodology documentation to clarify that false-verdict rates exceed ~11% when fewer than 5 tasks are evaluated and effects vary moderately or more (sd 0.30 or higher).

Fixed

  • Prevented automatic domain inference from using sentences from headingless skill body text as the domain.
  • Escaped Markdown formatting and user mentions (@, [, ], <, >, backticks, pipes, and newlines) in skill names and task excerpts in Markdown reports to prevent PR mention pings and table breakout.
  • Exited with a clear error when combining --markdown and --json flags instead of silently selecting JSON.

Dependencies

  • Updated dotenv to 18.0.5.

v0.13.0: Developer Habit Loop, Watch Mode, Pre-Commit Hook, Task Difficulty, and GFM Reports

Choose a tag to compare

@sx4im sx4im released this 01 Oct 13:41

Highlights of v0.13.0

  • Cascading Watch Mode (skillcheck watch <path>): Real-time file system monitoring with live token count delta tracking, instant background re-evaluation on save, and keyboard controls (r to re-run, q / Ctrl+C to exit).
  • Git Pre-Commit Regression Guard (skillcheck hook install / run): Evaluates staged skill files (git show :0:<path>) in temporary isolation to block regressions before commits land, failing closed on unexpected evaluation errors.
  • Task Difficulty Tuning (--difficulty <standard | hard | adversarial>): Calibrates generator prompts with multi-constraint edge cases and counter-intuitive boundary conditions to eliminate ceiling effects on frontier models (#68).
  • Smart Domain Extraction: Filters generic heading filenames (# CLAUDE.md, # AGENTS.md, # Instructions) to infer the genuine engineering domain (#69).
  • CI-Ready GitHub-Flavored Markdown Reports (--markdown): Exports GFM summary tables with escaped pipes and statistical breakdowns suitable for GitHub Actions step summaries and PR comments (#70).
  • Resumable Evaluations (--resume): Incremental atomic checkpointing preserving completed trial outputs, difficulty settings, and task suites across unexpected interruptions (#82, #88).
  • Deterministic Transcript Hashes: Canonical job sequencing (task -> trial -> arm) guaranteeing byte-identical reproducibility regardless of concurrent resolution order (#90).
  • Security Hardening: Strict allow-list parsing for local .env files, preventing hostile *_BASE_URL or configuration directory redirects from compromising shell API credentials.
  • Dashboard Leaderboard: Interactive leaderboard showcasing top featured agentic skills with live search and filter controls.

Verification

  • 39 test files · 280 tests passed (100% green)
  • TypeScript strict checking clean across source and tests
  • ESLint clean with zero warnings

v0.11.0

Choose a tag to compare

@sx4im sx4im released this 12 Sep 04:08

Added

  • Support standard --option=value arguments alongside space-separated options across all commands (e.g. --tasks=5, --trials=3, --output=result.json, and preserving equals signs in values/paths). (#53)
  • Zero-dependency cross-platform build script (scripts/build.mjs) replacing Unix-specific rm -rf / chmod shell pipelines, enabling native npm run build on Windows default shells (cmd.exe / PowerShell). (#54)
  • Windows CI smoke test job validating build and binary execution on windows-latest. (#54)

Changed

  • Consolidated NVIDIA NIM provider execution into OpenAiCompatClient with sendChatTemplateKwargs, eliminating the single-implementation NvidiaNimClient wrapper and redundant NvidiaConfig types. (#52)
  • Typed result card formatting directly against the EvalResult contract, removing defensive asRecord object guards and loose runtime fallbacks. (#55)
  • Tightened landing page hero heading to "Does your skill actually help?" for improved punchiness and cleaner 2-line layout. (#56)