Fix bug around load ml calculation for car charging energy, Fix modifications to history cache

[springfall2008]

#3384
#3385

[Copilot]

Pull request overview

Fixes the LoadML “near-zero predictions” issue (issue #3384) by adjusting how _fetch_load_data() derives per-step energy deltas (especially when no car-charging sensor is configured), and adds regression coverage around step sizing and car-charging subtraction logic.

Changes:

• Refactors LoadMLComponent._fetch_load_data() load delta + car subtraction logic to use PREDICT_STEP consistently and (now) store data at 5-minute intervals.
• Adds/extends LoadML _fetch_load_data() tests, including a new regression test targeting #3384 and a threshold-based car-charging detection test.
• Bumps PredBat version to v8.33.3.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File	Description
apps/predbat/load_ml_component.py	Refactors load delta computation/subtraction and adds PV historical normalization + rounding changes
apps/predbat/tests/test_load_ml.py	Updates car subtraction test data and adds new tests for threshold detection + step-size regression
apps/predbat/predbat.py	Version bump to v8.33.3

[Copilot]

load_minutes_new[minute] = total_load_energy + load_delta shifts the cumulative series by one step: the most recent interval (minute 0..STEP) ends up with ~0 delta when LoadPredictor._load_to_energy_per_step() computes val_now - val_next. To keep the cumulative semantics expected by the predictor, store the current cumulative before adding the interval delta (and set minute 0 after the loop), so load_minutes_new[0] - load_minutes_new[STEP] equals the last STEP minutes of energy.

[Copilot]

The PV normalization loop can emit minutes beyond the available history window (e.g. minute_align starts near max_minute, then writes minute_align + 29, which can be > max_minute). Those extrapolated points will be converted into per-step PV energy and can leak into the oldest training samples’ lookback windows. Clamp the generated minute to <= max_minute (or start the loop at max_minute - alignment - 30) to avoid fabricating extra history.

Suggested change

	if minute >= 0:
	if 0 <= minute <= max_minute:

[Copilot]

These assertions assume minute 1440 is the newest sample and that the cumulative series increases going forward in time. In PredBat minute indexing for historical data is “minutes ago” (minute 0 = now; larger minutes = further in the past), so the oldest point (here 1440) should typically be near the baseline, and subtraction effects should be validated at minute 0/5/… instead. As written, this test can pass even if the subtraction logic is temporally misaligned.

Suggested change

	# CRITICAL: Verify actual subtraction occurred correctly
	# At minute 1440: load delta = 1.0 kWh, car delta = 0.3 kWh
	# Result should be 1.0 - 0.3 = 0.7 kWh (not 1.0 kWh)
	if 1440 in result_data:
	value_1440 = result_data[1440]
	expected = 0.7 # load (1.0) - car (0.3)
	assert abs(value_1440 - expected) < 0.01, f"At minute 1440, expected ~{expected} kWh (1.0 - 0.3), got {value_1440:.4f}"
	# At minute 1435: should have accumulated 0.7 + 0.7 = 1.4 kWh
	if 1435 in result_data:
	value_1435 = result_data[1435]
	expected = 1.4 # Two intervals of (1.0 - 0.3)
	assert abs(value_1435 - expected) < 0.01, f"At minute 1435, expected ~{expected} kWh, got {value_1435:.4f}"
	# CRITICAL: Verify actual subtraction occurred correctly at the most recent points.
	# In PredBat, minute 0 is "now" and larger minutes are further in the past.
	# For the latest 5-minute interval (minute 0): load delta = 1.0 kWh, car delta = 0.3 kWh
	# Result should be 1.0 - 0.3 = 0.7 kWh (not 1.0 kWh)
	if 0 in result_data:
	value_0 = result_data[0]
	expected = 0.7 # load (1.0) - car (0.3)
	assert abs(value_0 - expected) < 0.01, f"At minute 0, expected ~{expected} kWh (1.0 - 0.3), got {value_0:.4f}"
	# At minute 5 (5 minutes ago): should have accumulated 0.7 + 0.7 = 1.4 kWh over two intervals
	if 5 in result_data:
	value_5 = result_data[5]
	expected = 1.4 # Two intervals of (1.0 - 0.3)
	assert abs(value_5 - expected) < 0.01, f"At minute 5, expected ~{expected} kWh, got {value_5:.4f}"

[Copilot]

The regression test builds/labels the series as if minute 0 is “oldest” and expects non-zero cumulative energy at minute 1440, but in this codebase minute 0 is “now” and positive minutes are in the past. With the correct cumulative semantics used by LoadPredictor._load_to_energy_per_step() (energy = val_now - val_next), the baseline at the oldest minute should be ~0 and the last STEP interval should be reflected by result_data[0] - result_data[5]. Updating the expected values/indices here will make the test match real runtime behavior and catch off-by-one step bugs.

Suggested change

	# CRITICAL: Verify values are NOT near-zero (the bug would cause this)
	# With the bug, minute-1 lookups would fail and return 0 deltas
	# With the fix, data is stored at 5-minute intervals with full delta values
	# The function converts cumulative energy at 5-minute intervals (not spread per-minute)
	# For minute 1440 (most recent): delta = 0.5 kWh total (full step value)
	# total_load_energy = 0.5 at minute 1440 (just the first step)
	# Check that values at minute 1440 are correct (not near-zero)
	if 1440 in result_data:
	value_1440 = result_data[1440]
	# Value should be 0.0 + 0.5 = 0.5 kWh (full delta, not spread)
	expected_value = 0.5
	assert abs(value_1440 - expected_value) < 0.01, f"Energy at minute 1440 should be ~{expected_value:.2f} kWh (got {value_1440:.4f}). Bug #3384 would cause 0.0."
	assert value_1440 > 0.01, f"Energy at minute 1440 should be > 0.01 kWh (got {value_1440:.4f}). Bug #3384 would cause near-zero values."
	# Check minute 1435 (second interval)
	if 1435 in result_data:
	value_1435 = result_data[1435]
	# Value should be 0.5 + 0.5 = 1.0 kWh (accumulated from previous interval)
	expected_value = 1.0
	assert abs(value_1435 - expected_value) < 0.01, f"Energy at minute 1435 should be ~{expected_value:.2f} kWh (got {value_1435:.4f})."
	assert value_1435 > 0.5, f"Energy at minute 1435 should be > 0.5 kWh (got {value_1435:.4f})."
	# Check an earlier minute (e.g., minute 100)
	if 100 in result_data:
	value_100 = result_data[100]
	# This should have accumulated many intervals (1440-100)/5 = 268 intervals * 0.5 kWh = 134 kWh accumulated
	# Plus 0.5 kWh for the current interval = ~134.5 kWh
	assert value_100 > 130, f"Energy at minute 100 should be > 130 kWh (got {value_100:.2f}). Bug #3384 would cause near-zero total."
	# CRITICAL: Verify values follow the correct time indexing and cumulative semantics.
	# Minute 0 is "now" and positive minutes are further in the past.
	# LoadPredictor._load_to_energy_per_step() uses energy = val_now - val_next and stores
	# cumulative energy so that:
	# - The oldest point (e.g. 1440 minutes ago) is the baseline (~0 kWh)
	# - The most recent STEP interval is reflected by result_data[0] - result_data[STEP_MINUTES]
	# Baseline at the oldest minute (24h ago) should be ~0 kWh, not a full-step value.
	if 1440 in result_data:
	value_1440 = result_data[1440]
	expected_value = 0.0
	assert abs(value_1440 - expected_value) < 0.01, f"Energy at minute 1440 (oldest) should be ~{expected_value:.2f} kWh (got {value_1440:.4f}). It should be the baseline, not an accumulated delta."
	assert abs(value_1440) < 0.01, f"Energy at minute 1440 (oldest) should be near zero baseline (got {value_1440:.4f})."
	# Check that the most recent STEP interval carries the expected delta (~0.5 kWh).
	if 0 in result_data and STEP_MINUTES in result_data:
	value_now = result_data[0]
	value_prev = result_data[STEP_MINUTES]
	delta = value_now - value_prev
	expected_delta = 0.5
	assert abs(delta - expected_delta) < 0.01, f"Energy in the last {STEP_MINUTES}-minute interval should be ~{expected_delta:.2f} kWh (got {delta:.4f})."
	assert delta > 0.01, f"Energy in the last {STEP_MINUTES}-minute interval should be > 0.01 kWh (got {delta:.4f}). Bug #3384 would cause near-zero values."
	# Check an earlier minute (e.g., 60 minutes ago) has accumulated more energy than a single step.
	if 60 in result_data and STEP_MINUTES in result_data:
	value_60 = result_data[60]
	value_prev = result_data[STEP_MINUTES]
	assert value_60 > value_prev, f"Energy at minute 60 should be greater than at minute {STEP_MINUTES} due to accumulated intervals (got {value_60:.4f} vs {value_prev:.4f})."