Skip to content

generate_terrain: map dask backends over an empty skeleton instead of the template#3575

Merged
brendancol merged 2 commits into
mainfrom
issue-3574
Jun 29, 2026
Merged

generate_terrain: map dask backends over an empty skeleton instead of the template#3575
brendancol merged 2 commits into
mainfrom
issue-3574

Conversation

@brendancol

Copy link
Copy Markdown
Contributor

Summary

  • The dask backends of generate_terrain regenerate every cell from coordinates and never read the template's values. They now map the per-chunk generator over a da.empty_like(data) skeleton instead of the template array, so a NaN-filled template (e.g. from_template()'s da.full(shape, nan)) drops out of the graph instead of being memset and discarded.
  • Removes the leftover data * 0 in the dask+cupy path, which built a NaN array the per-chunk cupy.zeros immediately discarded.

Output values are unchanged; this only removes wasted allocation and fill from the dask graph. Verified that the from_template -> generate_terrain graph no longer carries a full/NaN layer (only empty_like + xrspatial.terrain remain).

Closes #3574

Backends

  • numpy: unchanged (eager path allocates its own output)
  • cupy: unchanged
  • dask+numpy: skeleton swap
  • dask+cupy: skeleton swap + data * 0 removed

Test plan

  • test_terrain_dask_does_not_materialize_template (worley on and off): a poisoned template whose blocks raise if computed must still produce finite terrain
  • test_terrain_dask_skeleton_matches_numpy: skeleton output matches the numpy reference
  • test_terrain_dask_cupy_does_not_materialize_template
  • Full test_terrain.py suite green (56 passed, all four backends on the CUDA box)

The dask backends regenerate every cell from coordinates and never read
the template's values, but they mapped the per-chunk generator over the
template array itself. For a from_template() grid that meant computing a
da.full(shape, nan) and immediately discarding it. Map over a
da.empty_like skeleton instead so the template drops out of the graph.

Also removes the leftover data * 0 in the dask+cupy path, which built a
NaN array the per-chunk cupy.zeros discarded.

@brendancol brendancol left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR Review: generate_terrain dask backends map over an empty skeleton

Blockers

None.

Suggestions

  • The new tests cover worley on and off for the dask+numpy non-materialization case, but test_terrain_dask_cupy_does_not_materialize_template only runs with worley off. The worley pre-pass map_blocks was also switched to the skeleton, so a worley-on cupy case would exercise that second call site on GPU too. Low priority, since the numpy parametrize already covers both sites and the existing test_terrain_all_nan_template_dask_cupy_matches_numpy covers the default path. (xrspatial/tests/test_terrain.py)

Nits

  • The skeleton still produces one uninitialized np.empty/cupy.empty allocation per chunk feeding map_blocks. That is the expected pattern and there is no memset, so the real win is dropping the da.full(nan) fill, not the allocation. No code change needed.

What looks good

  • Verified the from_template -> generate_terrain graph drops the full/NaN layer (only empty_like + xrspatial.terrain remain) and the output stays finite and matches numpy.
  • Removes the leftover data * 0 in the dask+cupy path, the same NaN*0 pattern #3525 fixed on the eager backends.
  • The poison-template tests are a direct proof that the template is never read: blocks that raise on compute, yet the result computes fine.
  • empty_like preserves shape, dtype, and chunking, so block_info coordinates are unchanged and output values are identical (parity test passes).

Checklist

  • No algorithm change (output verified identical)
  • All four backends produce consistent results
  • NaN handling correct (template NaN dropped, output finite)
  • Edge cases covered (poison template, worley on/off)
  • Dask chunk boundaries handled correctly (empty_like preserves chunking)
  • Removes a premature materialization rather than adding one
  • Benchmark exists (benchmarks/benchmarks/terrain.py)
  • README feature matrix update not needed (no new function or backend)
  • Comments added; no public API change

@brendancol brendancol left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up review (after 8a9b1eb)

The earlier Suggestion is addressed: test_terrain_dask_cupy_does_not_materialize_template is now parametrized over worley_blend [0.0, 0.2], so the worley pre-pass map_blocks call site is exercised on the dask+cupy path as well.

  • Blockers: none
  • Suggestions: none outstanding
  • Nits: the earlier allocation note stands as documentation only; no code change warranted.

Local run: test_terrain.py template/materialize/skeleton tests pass (10 passed), full file green earlier (56 passed) on the CUDA box. Nothing new introduced by the follow-up.

@brendancol brendancol merged commit f3ab224 into main Jun 29, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

generate_terrain dask backends materialize the template's NaN fill before discarding it

1 participant