Summary
flow_accumulation_dinf has the same unbounded eager-allocation pattern that #1318 reported for flow_accumulation_d8. The numpy and cupy backends allocate H*W working arrays before any memory check runs, so a large raster requests tens of GB before failing.
The d8 fix landed in #1319. The dinf variant was deferred to a follow-up per the one-fix-per-security-PR policy, and this is that follow-up.
Allocations
_flow_accum_dinf_cpu (xrspatial/hydro/flow_accumulation_dinf.py, line 113):
| array |
dtype |
bytes/pixel |
| accum |
float64 |
8 |
| in_degree |
int32 |
4 |
| valid |
int8 |
1 |
| queue_r |
int64 |
8 |
| queue_c |
int64 |
8 |
CPU peak working set ~29 B/pixel, matching d8.
_flow_accum_dinf_cupy (line 333):
| array |
dtype |
bytes/pixel |
| accum |
float64 |
8 |
| in_degree |
int32 |
4 |
| state |
int32 |
4 |
GPU peak working set ~16 B/pixel, matching d8.
D-inf splits flow between two neighbours by a fractional weight, but those weights are computed inline from the angle and do not need their own H*W buffer, so the per-pixel cost is the same as d8.
Worked example
A 50000x50000 float64 dinf flow-direction raster on the numpy backend:
flow_dir already in RAM: 50000 * 50000 * 8 = 20 GB
- working set: 50000 * 50000 * 29 ~ 72.5 GB
A box with 64 GB of RAM raises an unhelpful allocator error (or starts swapping) instead of a clean MemoryError that points at the dask alternative.
Fix
Mirror #1319 with dinf-specific guards: _check_memory (29 B/pixel) on the numpy path and _check_gpu_memory (16 B/pixel) on the cupy path, both wired into flow_accumulation_dinf before backend dispatch. Dask paths skip the guard because per-tile allocations are bounded by chunk size.
The mfd variant has the same shape and will be handled in a separate follow-up PR.
Related: #1318, #1319.
Summary
flow_accumulation_dinfhas the same unbounded eager-allocation pattern that #1318 reported forflow_accumulation_d8. The numpy and cupy backends allocate H*W working arrays before any memory check runs, so a large raster requests tens of GB before failing.The d8 fix landed in #1319. The dinf variant was deferred to a follow-up per the one-fix-per-security-PR policy, and this is that follow-up.
Allocations
_flow_accum_dinf_cpu(xrspatial/hydro/flow_accumulation_dinf.py, line 113):CPU peak working set ~29 B/pixel, matching d8.
_flow_accum_dinf_cupy(line 333):GPU peak working set ~16 B/pixel, matching d8.
D-inf splits flow between two neighbours by a fractional weight, but those weights are computed inline from the angle and do not need their own H*W buffer, so the per-pixel cost is the same as d8.
Worked example
A 50000x50000 float64 dinf flow-direction raster on the numpy backend:
flow_diralready in RAM: 50000 * 50000 * 8 = 20 GBA box with 64 GB of RAM raises an unhelpful allocator error (or starts swapping) instead of a clean MemoryError that points at the dask alternative.
Fix
Mirror #1319 with dinf-specific guards:
_check_memory(29 B/pixel) on the numpy path and_check_gpu_memory(16 B/pixel) on the cupy path, both wired intoflow_accumulation_dinfbefore backend dispatch. Dask paths skip the guard because per-tile allocations are bounded by chunk size.The mfd variant has the same shape and will be handled in a separate follow-up PR.
Related: #1318, #1319.