v0.2.0 — Classifier mode

Skill Factory is on PyPI.

pip install "skill-factory[anthropic]"

New: classifier mode

Some skills are really classifiers: "is this skill malicious?", "does this PR do what it says?". With a decision model like jev, Skill Factory now trains the probe set (questions, framings, score, threshold, budget, policy) inside the same validation-gated loop:

skill-factory optimize -c examples/pr-mismatch/config.yaml   # kind: classifier
skill-factory lab freeze -c examples/pr-mismatch/config.yaml
skill-factory lab report -c examples/pr-mismatch/config.yaml --ask

• A lab that caches every answer. Up to 40 questions per request, offline rescoring and a spend cap.
• Recall at a false-positive budget, with constraints checked against the incumbent.
• Sealed test and held-out splits that open only after lab freeze, with every opening logged.
• Search and tuning. Selection and ensembling, reflective rounds, thresholds placed mid-gap between the classes, policy replay, re-ask determinism and recalibration.
• Exports to JSON, TypeScript and Python, with a SHA-256 and a byte-level parity check.
• metric.systemone, a cached yes/no rubric judge for skill mode.

Results

• jev (skill-scanner): from 29.0% of malicious skills caught at 1.28% false flags to 83.9% at 0.43% on validation, ahead of the hand-shipped 79.6% at 0.64%. It was rescored from 70k cached answers in 3 seconds with no new calls.
• pr-mismatch, live on jev-1.13.0: 3/3 hidden changes caught and 0/9 honest PRs flagged on the sealed test split, for $0.0025.

Full guide: docs/classifier.md · Changes: CHANGELOG

▶ The 61-second explainer (silent and narrated cuts) is attached below.