Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make use of Dask-Arrays in xcube.core.geom #672

Merged
merged 14 commits into from
Apr 28, 2022

Conversation

forman
Copy link
Member

@forman forman commented Apr 24, 2022

In this PR:

  • The functions mask_dataset_by_geometry() and rasterize_features() of module xcube.core.geom have been reimplemented to generate lazy dask arrays. Both should now be applicable to datasets that have arbitrarily large spatial dimensions. The spatial chunk sizes to be used can be specified using keyword argument tile_size.

  • Deprecated following functions of module xcube.core.geom:

    • is_dataset_y_axis_inverted() is no longer used;
    • get_geometry_mask() is no longer used;
    • convert_geometry() has been renamed to normalize_geometry().

Closes #593.
Closes #666.

Checklist:

  • Add unit tests and/or doctests in docstrings
  • Add docstrings and API docs for any new/modified user-facing classes and functions
  • New/modified features documented in docs/source/*
  • Changes documented in CHANGES.md
  • AppVeyor CI passes
  • Test coverage remains or increases (target 100%)

@codecov-commenter
Copy link

codecov-commenter commented Apr 24, 2022

Codecov Report

Merging #672 (1b1b1e7) into master (bcf0fb0) will increase coverage by 0.06%.
The diff coverage is 99.55%.

@@            Coverage Diff             @@
##           master     #672      +/-   ##
==========================================
+ Coverage   92.33%   92.39%   +0.06%     
==========================================
  Files         307      307              
  Lines       28844    28973     +129     
==========================================
+ Hits        26632    26769     +137     
+ Misses       2212     2204       -8     
Impacted Files Coverage Δ
xcube/core/geom.py 97.32% <99.15%> (+2.96%) ⬆️
test/core/test_geom.py 100.00% <100.00%> (ø)
xcube/core/timeseries.py 96.70% <100.00%> (ø)
xcube/core/schema.py 96.04% <0.00%> (+0.79%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bcf0fb0...1b1b1e7. Read the comment docs.

@forman forman changed the title Forman 666 rasterize features map blocks Make use of Dask-Arrays in xcube.core.geom Apr 25, 2022
@forman forman requested a review from pont-us April 25, 2022 05:55
@forman forman self-assigned this Apr 25, 2022
@pont-us
Copy link
Member

pont-us commented Apr 27, 2022

Looks good! Demo notebook works nicely, and I successfully ran a number of stress tests with larger data volumes.

@forman forman merged commit 6e37dec into master Apr 28, 2022
@forman forman deleted the forman-666-rasterize_features_map_blocks branch May 5, 2022 14:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

rasterize_features uses too much memory in xcube 0.10.2 Daskify geometry operations
4 participants