Skip to content

v0.2.0 - the toolkit can now test itself

Choose a tag to compare

@ChariereFiedler ChariereFiedler released this 21 Jul 09:15
· 11 commits to main since this release

The toolkit can now test itself, at every level: its code, its detection
scripts, and the behavior its prose skills are supposed to produce. Plus the
post-release audit fixes from a full-repo sweep.

Added

  • Skill evaluation harness (eval.ts): plays a skill through headless
    claude -p inside a throwaway fixture, then verdicts it with mechanical
    gates derived from the skill's own exit conditions, plus an adjunct LLM
    judge for what stays subjective (a judge verdict without a verbatim
    quote as evidence is downgraded to a failure). Verdicts: PASS, FAIL(gate),
    JUDGE-FAIL(criterion), ERROR(infra, never a regression).
  • An eval.yaml manifest for every skill (34/34), co-located with the
    skill. Closed gate vocabulary: file_exists, file_absent, grep_zero,
    grep_count, repo_clean, transcript_contains, transcript_absent,
    exit_ok. Manifests stay canonical and are never distributed to targets.
  • npm run eval:dry validates every manifest with no agent call, no network
    and no auth - wired into CI. Real runs (eval.ts run [--only <skill>]) are
    on demand and before a release, never per push.
  • Mutation harness (npm run test:mutation): 14 declared mutations must
    each turn their covering tests red; a surviving mutation fails CI.
  • install.ts --pick <token...> installs exactly the named artifacts.
  • Playground fixtures with planted defects and an EXPECTED.md ground truth:
    flawed-app, buggy-app, shipped-feature, design-system,
    audit-target.
  • CI matrix on Node 18 (the documented minimum) and 22.
  • docs/evaluating-skills.md.

Changed

  • Test harness split by domain (tests/) plus one co-located test procedure
    per executable skill script (skills/<skill>/scripts/*.test.ts); 80 tests.
    *.test.ts files are never distributed to target repos.
  • bin now points at install.ts. Anyone who linked the package got the
    deprecated rules.ts CLI before this release.
  • The interactive selector's keyboard logic is a pure, unit-tested reducer;
    the TTY shell only does I/O.
  • Em dashes replaced with plain punctuation across the toolkit.

Fixed

  • CI now triggers on main, the public default branch. It previously only
    listened to master and therefore never ran on the public repository.
  • Installer: pre-existing artifacts are backed up to *.pre-install.bak on
    first install instead of being silently overwritten (data loss); a
    settings.json whose hooks section has an unexpected shape no longer
    crashes mid-install; a truncated scan is announced instead of silently
    missing deep signals.
  • Audit suite: every domain scores with the flat mean defined in
    scoring-model.md (dead weight formulas removed); orchestrator
    contradictions resolved; performance-frontend restored in $ARGUMENTS.
  • worktree-env-setup hook: broken .env symlinks are repaired.
  • subscription-leak-scan.ts and templates/.adopted.example translated to
    English; docs no longer recommend the deprecated rules.ts sync;
    package.json description translated and engines declared.