Skip to content

flow_accumulation_dinf: no memory guard on H*W working arrays #1322

@brendancol

Description

@brendancol

Summary

flow_accumulation_dinf has the same unbounded eager-allocation pattern that #1318 reported for flow_accumulation_d8. The numpy and cupy backends allocate H*W working arrays before any memory check runs, so a large raster requests tens of GB before failing.

The d8 fix landed in #1319. The dinf variant was deferred to a follow-up per the one-fix-per-security-PR policy, and this is that follow-up.

Allocations

_flow_accum_dinf_cpu (xrspatial/hydro/flow_accumulation_dinf.py, line 113):

array dtype bytes/pixel
accum float64 8
in_degree int32 4
valid int8 1
queue_r int64 8
queue_c int64 8

CPU peak working set ~29 B/pixel, matching d8.

_flow_accum_dinf_cupy (line 333):

array dtype bytes/pixel
accum float64 8
in_degree int32 4
state int32 4

GPU peak working set ~16 B/pixel, matching d8.

D-inf splits flow between two neighbours by a fractional weight, but those weights are computed inline from the angle and do not need their own H*W buffer, so the per-pixel cost is the same as d8.

Worked example

A 50000x50000 float64 dinf flow-direction raster on the numpy backend:

  • flow_dir already in RAM: 50000 * 50000 * 8 = 20 GB
  • working set: 50000 * 50000 * 29 ~ 72.5 GB

A box with 64 GB of RAM raises an unhelpful allocator error (or starts swapping) instead of a clean MemoryError that points at the dask alternative.

Fix

Mirror #1319 with dinf-specific guards: _check_memory (29 B/pixel) on the numpy path and _check_gpu_memory (16 B/pixel) on the cupy path, both wired into flow_accumulation_dinf before backend dispatch. Dask paths skip the guard because per-tile allocations are bounded by chunk size.

The mfd variant has the same shape and will be handled in a separate follow-up PR.

Related: #1318, #1319.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions