v0.2.0 - the toolkit can now test itself
The toolkit can now test itself, at every level: its code, its detection
scripts, and the behavior its prose skills are supposed to produce. Plus the
post-release audit fixes from a full-repo sweep.
Added
- Skill evaluation harness (
eval.ts): plays a skill through headless
claude -pinside a throwaway fixture, then verdicts it with mechanical
gates derived from the skill's own exit conditions, plus an adjunct LLM
judge for what stays subjective (a judge verdict without a verbatim
quote as evidence is downgraded to a failure). Verdicts: PASS, FAIL(gate),
JUDGE-FAIL(criterion), ERROR(infra, never a regression). - An
eval.yamlmanifest for every skill (34/34), co-located with the
skill. Closed gate vocabulary:file_exists,file_absent,grep_zero,
grep_count,repo_clean,transcript_contains,transcript_absent,
exit_ok. Manifests stay canonical and are never distributed to targets. npm run eval:dryvalidates every manifest with no agent call, no network
and no auth - wired into CI. Real runs (eval.ts run [--only <skill>]) are
on demand and before a release, never per push.- Mutation harness (
npm run test:mutation): 14 declared mutations must
each turn their covering tests red; a surviving mutation fails CI. install.ts --pick <token...>installs exactly the named artifacts.- Playground fixtures with planted defects and an
EXPECTED.mdground truth:
flawed-app,buggy-app,shipped-feature,design-system,
audit-target. - CI matrix on Node 18 (the documented minimum) and 22.
docs/evaluating-skills.md.
Changed
- Test harness split by domain (
tests/) plus one co-located test procedure
per executable skill script (skills/<skill>/scripts/*.test.ts); 80 tests.
*.test.tsfiles are never distributed to target repos. binnow points atinstall.ts. Anyone who linked the package got the
deprecatedrules.tsCLI before this release.- The interactive selector's keyboard logic is a pure, unit-tested reducer;
the TTY shell only does I/O. - Em dashes replaced with plain punctuation across the toolkit.
Fixed
- CI now triggers on
main, the public default branch. It previously only
listened tomasterand therefore never ran on the public repository. - Installer: pre-existing artifacts are backed up to
*.pre-install.bakon
first install instead of being silently overwritten (data loss); a
settings.jsonwhosehookssection has an unexpected shape no longer
crashes mid-install; a truncated scan is announced instead of silently
missing deep signals. - Audit suite: every domain scores with the flat mean defined in
scoring-model.md(dead weight formulas removed); orchestrator
contradictions resolved;performance-frontendrestored in$ARGUMENTS. worktree-env-setuphook: broken.envsymlinks are repaired.subscription-leak-scan.tsandtemplates/.adopted.exampletranslated to
English; docs no longer recommend the deprecatedrules.ts sync;
package.jsondescription translated andenginesdeclared.