Skip to content

prediction error

Latest

Choose a tag to compare

@samueleternity samueleternity released this 25 Oct 11:30
0d31691
Add Surprise-Gated Recall Phase 1 (#40)

* Add Surprise-Gated Recall Phase 1

Add a seven-lesson SGR dataset, generator validity checks, training and evaluation diagnostics, and an optional next-token prediction head using split-graph backbone features. Document Phase 0 and Phase 1 usage.

* Trace split-graph forward instability

Add targeted and on-error numeric tracing for split-graph forward boundaries, with CSV output and failed-batch replay. Prevent duplicate CfC residuals and accumulate the graph-level residual in fp32 to reduce overflow risk.

* Add console summaries for SGR diagnostics

Print run-tagged training, phase-one, and evaluation metrics alongside existing logs, including a compact episode-grouped trace of sampled token positions.

* Normalize Mamba outputs and ease lesson gates

Add non-affine output normalization to the parallel Mamba backbone, applying it after workspace deltas to preserve the standard forward path. Lower graph traversal and surprise recall lesson advancement thresholds from 85% to 80%.

* Add SGR error-gated memory and workspace paths

Add isolated Phase 2 backbone-only SGR arms and Phase 4 prediction-error consumers for DNC writes, workspace writes, and reconsideration. Include signal diagnostics, evaluation controls, and lower the surprise-recall lesson advancement threshold.

* Add split-graph backbone-only mode

Allow split-graph runs without DNC memory or its sequential combiner, while preserving backbone MoE, workspace, training, and checkpoint support. Also forward elapsed-time spans to CfC stages and keep Phase 2 Mamba residuals in fp32.

* Prevent NaNs in surprise recall paths

Run the prediction head in float32, compute loss only for valid stream positions, and mask excluded positions with `masked_fill` so NaNs cannot propagate through zero multipliers.

* Stabilize Mamba residuals and configure log intervals

Bound mixer residual RMS across Mamba paths while keeping residual streams in FP32, and apply the configured MoE residual scale to workspace specialist candidates. Add a configurable training summary interval via `--log-every`.

* Linted, formatted + a few bug fixes.

* Harden masked losses and add regression coverage

Select valid positions before cross-entropy to prevent masked non-finite logits from contaminating losses, and return differentiable zeros for empty selections. Add coverage for surprise-recall generation, next-token signals, numerical stability, and workspace diagnostics; update related documentation.

* test and linting fixes