perf(load_ml): halve the memory of a training pass

[springfall2008]

Problem

Profiling a plan cycle found ML training dominating peak memory at 1.4 GB RSS, with 81% of the live heap in the training thread. A curriculum pass over three weeks of history builds 11,500 samples × 1,446 float32 features — 66.7 MB — and the training path held several copies of it simultaneously.

Four changes

1. Preallocate the feature matrix. _create_dataset built a Python list of 11,500 separate per-sample arrays, then copied it into np.array() — both alive at once. Rows now go straight into a preallocated matrix, which also removes the five temporary arrays plus the concatenated result that _build_sample created per sample (1.28M allocations each in the profile).

2. Normalise in place. _normalize_features gains an opt-in in_place flag, used at the three training sites where the caller is provably finished with the array — only the _norm arrays are read from there on. The default still copies, so the prediction paths are untouched.

3. Compute statistics over row blocks. np.std(X, axis=0) materialises the deviations array internally — another full copy of the matrix to produce 1,446 numbers. _feature_mean_std accumulates over blocks in float64: no temporary larger than one block, and considerably more accurate:

	before (float32 accum)	after (float64 accum)
fitted mean error	2.2e-04	1.5e-05 (float32 representation floor)
fitted std error	2.4e-06	4.6e-10

4. Stop promoting the matrix to float64. _get_min_std_array returned a float64 array, so np.maximum made feature_std float64 and (X - mean) / std promoted the entire normalised matrix to double precision — 133.3 MB rather than 66.6 MB — running every forward and backward pass in float64 against float32 weights. This looks accidental: the dataset, weights and activations are all deliberately float32.

Measured impact

End to end, a real curriculum training run against live Home Assistant history, both arms completing 9 passes with zero exceptions on matched data:

	before	after
peak RSS	1440.1 MB	811.4 MB (−43.7%)
settled RSS	906.4 MB	614.9 MB (−32.2%)

At component level, through _create_dataset + _normalize_features at 11,519 samples:

	before	after
dataset construction	181.8 MB	107.2 MB
peak growth	298.9 MB	85.9 MB (−71%)
normalised matrix	float64, 133.3 MB	float32, 66.6 MB

(Peak growth was 165.1 MB before review feedback; squaring the deviations in place and shrinking the statistics block took a further 79 MB.)

Numerics

Change 4 shifts results, so this is not a pure refactor. All 29/29 ML tests pass before and after.

Quantifying the shift is harder than it first appears: the ML suite is not deterministic. Two consecutive runs of identical code produce different val_mae sequences. Against that noise floor:

comparison	mean abs diff	max
same code, two runs (noise)	0.00160	0.01890
original vs changed code	0.00546	0.03920

So the change is real at roughly 3x the noise, but a single before/after run cannot pin the magnitude more precisely than that. Changes 1-3 are provably behaviour-preserving; change 4 is the only one that moves results.

The feature matrix itself is unchanged — a checksum over it is pinned in the new tests and does not move, and that check is deterministic.

A bug this introduced, and caught

The second commit fixes a defect from change 1. Making _build_sample write into a destination row and return just the target left its three early returns on the old two-value form, so on a gap in the history it returned the tuple (None, None) — and if target is None is false for a tuple. Gap rows were counted as valid and the tuple landed in the target list, so every curriculum pass threw ValueError: setting an array element with a sequence and the model never trained.

It was invisible from the test suite because every fixture builds unbroken synthetic history, and invisible at runtime because _do_training catches the exception, logs it, and retries a minute later — Predbat kept planning normally while the forecaster silently never produced a model. It surfaced only from the end-to-end profile: 9 curriculum passes before the change, 0 after.

The new test punches two sensor-dropout holes in the history and asserts the dataset stays dense, its targets and weights stay aligned, and every feature row is fully written.

Testing

• tests/test_ml_memory.py, 20 assertions across 6 tests, registered as ml_memory: normalisation numerics, in-place behaviour and default non-mutation, dtype contract, a _create_dataset characterisation test pinning the feature-matrix checksum and sample count, statistics accuracy on a large-mean dataset, and gap handling
• Each test verified failing before its change — the accuracy test was re-verified RED after its threshold was corrected to the float32 representation floor, so the fix was not graded against a moved goalpost
• ./run_all --quick — all tests pass, 20/20 random scenarios match baseline across 320 fields
• ./run_all --test load_ml — 29/29 pass
• ./run_pre_commit — all hooks pass
• End-to-end curriculum training verified: 9 passes, 0 exceptions, matching unmodified behaviour

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR reduces peak RSS during the ML load-forecaster training path by eliminating avoidable copies of the feature matrix and preventing accidental float64 promotion, while adding targeted regression tests to pin numerics and dtype/memory contracts.

Changes:

• Preallocates and fills the training/validation feature matrices row-by-row in _create_dataset() to avoid list-of-rows + np.array() double-allocation.
• Adds block-wise float64 accumulation for mean/std fitting and an optional in-place normalization path used by training.
• Ensures normalization stays float32 end-to-end (fixing _get_min_std_array() dtype), and adds new ml_memory unit tests.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
apps/predbat/load_predictor.py	Memory-focused refactor of dataset construction and normalization; adds block-wise stats and float32 min-std to avoid float64 promotion.
apps/predbat/tests/test_ml_memory.py	New tests pinning dataset checksum/sample count, normalization numerics, in-place behavior, dtype contract, and stats accuracy.
apps/predbat/unit_test.py	Registers the new ml_memory test group in the unit test runner.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[springfall2008]

Thanks — both comments addressed.

_build_sample early returns (load_predictor.py:633) — correct, and this was a real failure, not a latent one. Every curriculum pass threw ValueError: setting an array element with a sequence and the model never trained. It was invisible from the test suite because every fixture builds unbroken synthetic history, and invisible at runtime because _do_training catches the exception and retries a minute later, so Predbat kept planning normally while the forecaster silently produced nothing.

Fixed in 6e7e9be, with a test that punches two sensor-dropout holes in the history and asserts the dataset stays dense and aligned. Verified end to end: 9 curriculum passes complete with no exceptions, matching unmodified behaviour.

np.square temporary (load_predictor.py:846) — also correct, and bigger than it looks. At the previous block=4096 each buffer was 47.4 MB, so the pair came to 94.8 MB of temporaries to compute statistics over a 66.6 MB matrix — more than the matrix itself.

Fixed in c58fd52: squaring into the deviation buffer (bitwise identical), and dropping the block to 512 so the buffer is 5.9 MB rather than 47.4 MB. The smaller block regroups the float64 summation, moving the accumulated std by ~5e-16 — eleven orders below the float32 resolution of the stored result — so both statistics are bitwise identical once cast. Verified directly rather than assumed.

Peak growth of _create_dataset + _normalize_features at 11,519 samples:

	peak growth
before any of this work	298.9 MB
after the original PR	165.1 MB
after this review	85.9 MB (−71%)

Separately, a correction to this PR's own description. I originally cited a val_mae before/after comparison as evidence of how much the float32 change moves results. Checking that method, the ML suite is not deterministic — two consecutive runs of identical code give different values (mean |diff| 0.00160, max 0.01890). The before/after difference is 0.00546 mean / 0.03920 max, so the shift is real at roughly 3x the noise floor, but the precise figure I quoted was contaminated by run-to-run variation, and my claim that early epochs were "consistently better" was reading noise. The description has been corrected.