Skip to content

v3.1.0 — Gated self-update: held-out gate + edit budget (SkillOpt core)

Latest

Choose a tag to compare

@smixs smixs released this 21 Jul 02:45

Borrows the core discipline of microsoft/SkillOpt — validation-gated bounded text edits — into skill-conductor's IMPROVE mode. No SkillOpt dependency; the methodology is implemented natively.

Improved

  • Held-out gate for body edits (Mode 2 → Gated Self-Update Loop). Evals are split once per session into train/held-out (scripts/split_evals.py: deterministic, seed 42, stratified by the new optional evals[].category). Lessons and edits are formed from TRAIN failures only; a candidate is accepted iff (a) no held-out assertion flips pass→fail (with a confirmation re-run for noise), (b) train pass-rate strictly improves, (c) no new critical BinEval failure. This closes the overfitting hole where edits were accepted by the same evals that produced them.
  • Edit budget — at most 3 atomic edits per iteration, each labeled with its lesson. No wholesale rewrites, so a gate rejection stays attributable to a single edit.
  • Case transitions — assertion-level diff parent→candidate (improved / regressed / persistent-fail / stable-success), recorded as an additive transitions block in benchmark.json.

Fixed

  • split_eval_set was private to run_loop.py (Mode 5 OPTIMIZE, descriptions only) — generalized into utils.split_evals(stratify_key=...) and reused for body-edit gating. The historical should_trigger split is preserved bit-for-bit (verified on 60 configurations).

Field-tested

Validated end-to-end on humanizer-ru (15 evals, 103 assertions): baseline failure found on train, 3 gated edits accepted — train 62/63 → 63/63, held-out 36/40 → 37/40, zero regressions, one train-derived lesson transferred to a held-out case.