Guard flow_accumulation_dinf() against unbounded memory allocations (#1322)#1325
Merged
brendancol merged 1 commit intomainfrom Apr 29, 2026
Merged
Guard flow_accumulation_dinf() against unbounded memory allocations (#1322)#1325brendancol merged 1 commit intomainfrom
brendancol merged 1 commit intomainfrom
Conversation
This was referenced Apr 29, 2026
brendancol
added a commit
that referenced
this pull request
Apr 29, 2026
) Add memory guards to the numpy and cupy dispatch branches. CPU peak working set is ~33 B/pixel: float64 cast (8) + labels (8) + state (1) + path_r (8) + path_c (8). A 50000x50000 raster needs ~83 GB before the dispatch even runs. GPU peak is ~28 B/pixel on the device: flow_dir_f64 (8) + pp_f64 (8) + labels (8) + state int32 (4). Helpers _check_memory and _check_gpu_memory raise MemoryError with the grid dimensions and a pointer to the dask backend when the projected working set exceeds 50% of available memory. Dask paths skip the guard since per-tile allocations are bounded by the user's chunk size. Same pattern as #1319, #1325, #1324, #1326.
brendancol
added a commit
that referenced
this pull request
Apr 29, 2026
…1332) Adds a memory guard to flow_length_d8() matching the pattern from #1319, #1324, #1325, and #1326. The eager numpy and cupy backends now raise MemoryError before allocating an HxW working set that would exceed 50% of available host or GPU memory. Dask paths skip the check since per-tile allocations are bounded by chunk size. CPU budget is 29 B/px (in_degree int32 + valid int8 + flow_len float64 + order_r/order_c int64). GPU budget is 32 B/px covering the device input + output copies in _flow_length_cupy.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
_check_memory/_check_gpu_memorybudget checks (29 B/px CPU, 16 B/px GPU, 50% threshold) to the eager numpy and cupy backends inflow_accumulation_dinf.py.Fixes #1322.
Background
_flow_accum_dinf_cpuallocatesaccum(float64),in_degree(int32),valid(int8), and twoH*Wint64 BFS queues, ~29 B/pixel of working memory plus the caller's input._flow_accum_dinf_cupyallocatesaccum(float64) +in_degree(int32) +state(int32) ~16 B/pixel of GPU memory. Neither backend checked against available memory, so a 50000x50000 numpy raster asked for ~72 GB of host memory before anything errored out.D-inf splits flow between two neighbours by a fractional weight, but the weights are computed inline from the angle and do not need their own H*W buffer, so the per-pixel cost matches the d8 BFS kernel.
Same root cause and fix shape as #1318 / #1319. The mfd variant has the same shape and will be handled in a separate follow-up PR per the one-fix-per-security-PR policy.
Test plan
pytest xrspatial/hydro/tests/test_flow_accumulation_dinf.py-- 23 passed (19 existing + 4 new memory-guard cases)MemoryErrorwith a "working memory" / dimension messagepytest xrspatial/hydro/tests/-- 635 passed (full hydro suite)