Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make processing dask-friendly to allow larger-than-memory and parallel processing #25

Closed
1 of 4 tasks
alexamici opened this issue May 17, 2022 · 0 comments · Fixed by #29
Closed
1 of 4 tasks
Assignees
Labels
enhancement New feature or request

Comments

@alexamici
Copy link
Member

alexamici commented May 17, 2022

Almost all operations can be chunked via the DEM chunks (not the image ones).

This includes:

  • allow controlling the DEM chunks
  • use dask-friendly xarray-sentinel helpers
  • make internal functions (that need it) dask-friendly via xarray.map_blocks
  • add borders to the gamma flattening processing to avoid "seeing" the borders of the blocks

The main source of dask-unfriendliness is the wide spread use of DataArray.interp with the target values on the DEM coordinates. All operations (except the gamma flattening) can be performed on ("x", "y") blocks exactly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
1 participant