Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Daskify geometry operations #593

Closed
forman opened this issue Jan 26, 2022 · 2 comments · Fixed by #672
Closed

Daskify geometry operations #593

forman opened this issue Jan 26, 2022 · 2 comments · Fixed by #672
Assignees
Labels
enhancement New feature or request

Comments

@forman
Copy link
Member

forman commented Jan 26, 2022

Is your feature request related to a problem? Please describe.

The xcube function xcube.core.geom.rasterize_features() seems to eagerly load large numpy arrays into memory often resulting in an OOM error.

xcube.core.geom.get_geometry_mask() may be affected too as it is effectively a wrapper for rasterio.features.geometry_mask() which may not use Dask at all (to be verified!).

Describe the solution you'd like

Instead they should produce lazy dask arrays.

@forman forman self-assigned this Jan 26, 2022
@forman
Copy link
Member Author

forman commented Jan 26, 2022

...which may not use Dask at all (to be verified!).

Verified, it does not use Dask at all and returns a numpy array. From rasterio docstring:

This function requires significant memory resources. The shapes
iterator will be materialized to a Python list and another C copy of
that list will be made. The `out` array will be copied and
additional temporary raster memory equal to 2x the smaller of `out`
data or GDAL's max cache size (controlled by GDAL_CACHEMAX, default
is 5% of the computer's physical memory) is required.

@forman forman added in progress The assignee is working on it enhancement New feature or request labels Jan 26, 2022
@forman forman removed the in progress The assignee is working on it label Jan 27, 2022
@forman
Copy link
Member Author

forman commented Jan 27, 2022

Issue has been party addressed in #594.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant