perf(prediction): build the kernel's ctypes arrays via array.array (1.9x faster planning)

[springfall2008]

Summary

After #4505, #4507 and #4508, marshalling into the C++ kernel was the largest remaining cost in the planner — 8.1s of a 13.9s profiled plan, more than everything else combined:

ncalls  tottime  function
 42747    4.729   prediction_kernel.py:455(run_prediction_kernel)   ← the six list comprehensions
171019    2.194   prediction_kernel.py:268(int32_array)
 86052    1.177   prediction_kernel.py:263(double_array)

The helpers built their buffers as (ctypes.c_double * n)(*values), which unpacks the list into positional arguments. Copying from an array.array of the same type is several times faster. Measured on the shapes the planner actually passes (266 charge windows):

approach	µs/call
current (c_int32 * n)(*values)	35.00
array.array + from_buffer	12.68
content-keyed memoisation	10.84

Alternatives measured and rejected

• Memoising the geometry arrays (10.8µs) is only marginally ahead, because the content key still has to be built and hashed — the very work that makes the naive version slow. It would also have to stay correct across the passes that mutate window bounds in place (optimise_swap_export and friends), which is a poor trade for 1.8µs.
• Reusing the soc_out buffer — allocation is 0.15µs/call, not worth it.

Safety

from_buffer returns a view over the array.array, not a copy, so the backing object must outlive the kernel call — the failure mode is reading freed memory, silently and intermittently. ctypes keeps it alive through the view's _objects; a new test asserts that rather than assuming it, and also covers value round-trip and the empty-array case.

The pool workers are separate forked processes (multiprocessing.Pool, set_start_method("fork") in hass.py), so no buffer is ever shared between workers. Verified by running the same plan single-process and with a 4-process pool and confirming identical charge/export windows and limits — not added as a test since it needs two full plan runs.

Impact

	before	after
worst benchmark scenario	11.03s	8.39s
mean optimise time, 20 scenarios	3.717s	1.961s (1.9x)
plan metric / cost, all 20	—	identical

Cumulatively across #4505, #4507, #4508 and this: the worst scenario is down from 152.0s to 8.39s (18x), and the benchmark mean from 13.674s to 1.961s.

Test plan

• New run_marshalling_tests in the kernel parity suite: value round-trip, _objects retention, empty array
• kernel_parity (450 random configurations + 250 clipping layouts) passes
• Pool path verified identical to single-process
• run_random: metric and cost identical on 20/20
• --quick, debug_cases, pre-commit all pass

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR speeds up the prediction planner’s Python→ctypes marshalling path by building ctypes arrays via array.array + from_buffer, reducing per-simulation overhead before calling the C++ kernel. It also updates kernel parity tests and refreshes benchmark result fixtures to reflect the improved runtimes.

Changes:

• Switch double_array/int32_array marshalling helpers to array.array(...).from_buffer(...) for faster ctypes buffer creation.
• Add kernel parity “marshalling” tests to validate value round-trip and backing-buffer retention semantics.
• Update coverage/cases/random_results.json with new run timestamp and improved runtime_s numbers.

Reviewed changes

Copilot reviewed 2 out of 3 changed files in this pull request and generated 3 comments.

File	Description
apps/predbat/prediction_kernel.py	Uses array.array + from_buffer to construct ctypes arrays faster for kernel inputs.
apps/predbat/tests/test_kernel_parity.py	Adds parity-suite tests to validate marshalling safety/behaviour.
coverage/cases/random_results.json	Updates benchmark fixture metadata and runtimes after the perf change.

Suppressed comments (1)

apps/predbat/prediction_kernel.py:281

• int32_array() uses array.array('i'), where 'i' is a C "int" and its item size is platform-dependent. That can break the function’s implied int32 contract (ctypes.c_int32) on platforms where C int is not 32-bit, and from_buffer would then interpret the backing bytes incorrectly. Add an itemsize guard with a safe fallback.

    """Create a ctypes int32 array from a Python list - see double_array for why array.array is used"""
    backing = array.array("i", values)
    return (ctypes.c_int32 * len(backing)).from_buffer(backing)

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[springfall2008]

Fixed in e10031c. This one was worth more than robustness on unusual platforms - I checked what actually happens on a mismatch and it is silent, not an exception: building a c_int32 array over an array('d') backing returns [0, 1072693248, 0] rather than raising, because from_buffer only checks the buffer is large enough. So a wrong-width typecode would feed corrupted window bounds into the kernel. The typecode is now selected at import by matching itemsize against the ctypes element, with a fallback to the original (ctypes.c_double * n)(*values) construction when nothing matches - such a platform loses the speed-up instead of silently corrupting results. Covered by a new itemsize assertion in the marshalling test.