Borrows the core discipline of microsoft/SkillOpt — validation-gated bounded text edits — into skill-conductor's IMPROVE mode. No SkillOpt dependency; the methodology is implemented natively.
Improved
- Held-out gate for body edits (Mode 2 → Gated Self-Update Loop). Evals are split once per session into train/held-out (
scripts/split_evals.py: deterministic, seed 42, stratified by the new optionalevals[].category). Lessons and edits are formed from TRAIN failures only; a candidate is accepted iff (a) no held-out assertion flips pass→fail (with a confirmation re-run for noise), (b) train pass-rate strictly improves, (c) no new critical BinEval failure. This closes the overfitting hole where edits were accepted by the same evals that produced them. - Edit budget — at most 3 atomic edits per iteration, each labeled with its lesson. No wholesale rewrites, so a gate rejection stays attributable to a single edit.
- Case transitions — assertion-level diff parent→candidate (improved / regressed / persistent-fail / stable-success), recorded as an additive
transitionsblock in benchmark.json.
Fixed
split_eval_setwas private torun_loop.py(Mode 5 OPTIMIZE, descriptions only) — generalized intoutils.split_evals(stratify_key=...)and reused for body-edit gating. The historical should_trigger split is preserved bit-for-bit (verified on 60 configurations).
Field-tested
Validated end-to-end on humanizer-ru (15 evals, 103 assertions): baseline failure found on train, 3 gated edits accepted — train 62/63 → 63/63, held-out 36/40 → 37/40, zero regressions, one train-derived lesson transferred to a held-out case.