Skip to content

flow_length_dinf: no memory guard on H*W working arrays #1355

@brendancol

Description

@brendancol

flow_length_dinf in xrspatial/hydro/flow_length_dinf.py allocates several full-grid working arrays in its eager numpy and cupy backends with no upfront budget check. A 50000x50000 raster asks for about 72 GB of host RAM before anything errors out.

Same shape as #1332 for flow_length_d8 and #1351 for flow_length_mfd, and the broader hydro audit started with #1318/#1319 for flow_accumulation_d8. Eager backends check, dask backends skip since per-tile allocations are bounded by chunk size.

Allocations

_flow_length_dinf_downstream_cpu and _flow_length_dinf_upstream_cpu (called by both numpy and cupy backends) allocate:

Array dtype bytes/px
in_degree int32 4
valid int8 1
flow_len float64 8
order_r int64 8
order_c int64 8
Total 29

The D-inf flow direction array is 2-D (angles in radians), matching the d8 budget. The numpy public dispatch also runs fd = data.astype(np.float64) before the kernel, but that 8 B/px copy is dwarfed by the 29 B/px kernel locals.

For the cupy backend, _flow_length_dinf_cupy calls .get().astype(np.float64) then runs the CPU kernel and copies the output back via cp.asarray. Device-side residency at peak is the input float64 (8 B/px) plus the output float64 (8 B/px); host-side matches the 29 B/px CPU budget. Use 32 B/px as a conservative GPU budget covering both copies plus headroom.

Worked example

50000x50000 = 2.5e9 pixels. CPU peak working set:

2.5e9 * 29 = 72.5 GB

Allocated before any sanity check runs.

Fix

Mirror PR #1332: _check_memory and _check_gpu_memory helpers, wired into the numpy and cupy dispatch in flow_length_dinf(). Leave dask alone. Add tests for oversize rejection, valid pass-through, dask bypass, and error message content.

One fix per PR per the security-sweep policy.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinghigh-priorityoomOut-of-memory risk with large datasets

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions