Skip to content

Guard hand_dinf() against unbounded memory allocations (#1344)#1346

Merged
brendancol merged 1 commit intomainfrom
issue-1344-hand-dinf-memory-guard
Apr 29, 2026
Merged

Guard hand_dinf() against unbounded memory allocations (#1344)#1346
brendancol merged 1 commit intomainfrom
issue-1344-hand-dinf-memory-guard

Conversation

@brendancol
Copy link
Copy Markdown
Contributor

Closes #1344.

Same shape as #1326 (hand_d8) and #1340 (hand_mfd). The numpy and cupy backends of hand_dinf both run _hand_dinf_cpu, which allocates seven full-grid working arrays totalling 38 bytes/pixel. A 50000x50000 raster would ask for ~95 GB before any sanity check ran.

Allocations

_hand_dinf_cpu allocates:

Array dtype bytes/px
in_degree int32 4
valid int8 1
is_stream int8 1
drain_elev float64 8
hand_out float64 8
order_r int64 8
order_c int64 8
Total 38

The cupy path copies its three inputs to host via .get() then runs the same CPU kernel, so it also pays 38 B/px on the host plus the input copies. A conservative 32 B/px applies on the device side for symmetry with #1326 and #1340.

Fix

Adds _check_memory and _check_gpu_memory helpers and wires them into the numpy and cupy branches of hand_dinf(). Dask paths are untouched since per-tile allocations are bounded by chunk size.

Tests

Five new tests cover:

  • numpy oversize raster raises MemoryError
  • numpy normal raster passes the guard
  • dask path skips the guard even when memory is reported as 1 byte
  • error message includes grid dimensions and a "use dask" hint
  • cupy oversize raster raises MemoryError (CUDA-only)

Full hydro suite runs 635 passed, 1 failed (test_stream_link_dask_temp_cleanup, a known flake unrelated to this change).

Mirrors the fix from #1326 (hand_d8) and #1340 (hand_mfd). The numpy
and cupy backends call _hand_dinf_cpu which allocates seven full-grid
working arrays totaling 38 bytes/pixel; a 50000x50000 raster would
ask for ~95 GB before any sanity check ran.

Adds _check_memory and _check_gpu_memory helpers, wired into the
numpy and cupy dispatch in hand_dinf(). Dask backends are unchanged
since per-tile allocations are bounded by the user's chunk size.

Five new tests cover oversize rejection on numpy and cupy, valid
pass-through, dask bypass, and error message content.
@github-actions github-actions Bot added the performance PR touches performance-sensitive code label Apr 29, 2026
@brendancol brendancol merged commit 2367bdc into main Apr 29, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

performance PR touches performance-sensitive code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

hand_dinf: no memory guard on H*W working arrays

1 participant