#4 M0 predictability harness → GATE: FAIL - #4
Merged
Merged
Conversation
harness/m0_gate.py reconstructs per-(source,gen,indiv) trajectories from sensor
CSVs (.gz supported), builds 32-step context windows, and fits six predictors of
s_{t+1} — persistence, per-sensor mean, ridge (s_t only), MLP, GRU, and a ctx-32
transformer (d64/4L/4H) — scoring PER-SENSOR R^2 on a held-out split of whole
trajectories. All predictors share the identical (context, target) test set, so
dR^2 over persistence is apples-to-apples.
The gate (SPEC §2, fixed): the transformer must beat persistence by mean dR^2 >
0.05 AND on >= 6 of the 11 non-trivial sensors. Aggregate R^2 is never reported —
AGE/OSC1 are free ~1, RANDOM free ~0, LOC_*/BOUNDARY_* already near-perfect under
persistence. Zero-variance gate sensors are reported as `const` (persistence exact,
dR^2=0, non-winnable) rather than crashing on 0/0.
The three NN models train on per-sensor-standardized targets (equal loss weight)
and are scored on the raw scale, so low-variance sensors like POPULATION get a fair
fit rather than being ignored by the optimizer.
- harness/test_m0_gate.py — 9 unit tests (R^2 edge cases, window alignment, gate
math incl. a PASS case and constant-sensor handling). All pass.
- scripts/build_report.py — renders gate JSON(s) into a self-contained HTML report.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ran the harness on the #3 evolved population (5 seeds, gen 119, 87 200 windows, 17 440 held out) and a gen-0 contrast. GATE: FAIL — mean ΔR² = -0.0019 over the 11 non-trivial sensors (threshold > 0.05), transformer beats persistence on 1/11 (need >= 6). The criterion was NOT modified. Why: persistence already wins. 4 of the 11 gate sensors are dead constants (BARRIER_FWD/LR and LONGPROBE_BAR_FWD — no barriers; SIGNAL0 — evolution selects signal emission away), and the other 7 already sit at persistence R² ~0.99, leaving no headroom. The gate also fails at gen 0, so the verdict is structural, not an artifact of the (weak) evolution observed in #3. - RESULTS.md — the number, what it means, robustness, caveats, and the recommended world-enrichment lever (moving barriers via replaceBarrierType, after restoring Miller-level mutation) for issue #5. - data/gate_gen119.json, data/gate_gen0.json — full per-sensor records. - data/m0_report.html — self-contained per-sensor table + verdict (both gens). - README: M0 status = FAIL. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
schmug
force-pushed
the
issue-3-longrun-capture
branch
from
July 14, 2026 22:07
268aa50 to
d4d8873
Compare
schmug
force-pushed
the
issue-4-m0-harness
branch
from
July 14, 2026 22:07
a363df2 to
d985d5b
Compare
This was referenced Jul 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#4 — M0 predictability harness → GATE: FAIL
harness/m0_gate.py: per-(source,gen,indiv)trajectories → 32-step windows → sixpredictors of
s_{t+1}(persistence, mean, ridge, MLP, GRU, ctx-32 transformer),scored per-sensor R² on held-out trajectories. All models share the identical
(context,target) test set; NN models train on per-sensor-standardized targets for a
fair fit. Aggregate R² is never reported; constant gate sensors are
const(ΔR²≡0).GATE: FAIL — on the evolved gen-119 population (5 seeds): mean ΔR² = −0.0019
over the 11 non-trivial sensors (need > 0.05), transformer wins 1/11 (need ≥ 6).
The criterion was not modified.
Why: persistence already wins. 4 of 11 gate sensors are dead constants
(BARRIER_FWD/LR, LONGPROBE_BAR_FWD — no barriers; SIGNAL0 — evolution selects
emission away); the other 7 sit at persistence R² ≈ 0.99. Fails at gen 0 too, so the
verdict is structural. See
RESULTS.mdanddata/m0_report.html.Acceptance ✅
GATE:line.sensors_gen0_gen4.csvsmoke fixture without error.python3 harness/test_m0_gate.py→ 9/9 (incl. a PASS case and constant handling).RESULTS.md: the number, what it means, and the recommended first world-enrichmentlever (moving barriers via
replaceBarrierType, after restoring Miller-levelmutation) for #5.