-
Notifications
You must be signed in to change notification settings - Fork 295
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Convert AHI HSD dask chunking to be based on band resolution #2584
Conversation
Oh I still need to add tests for the reader chunking and dtype fixes there. |
Codecov Report
@@ Coverage Diff @@
## main #2584 +/- ##
=======================================
Coverage 94.92% 94.92%
=======================================
Files 350 350
Lines 51085 51098 +13
=======================================
+ Hits 48491 48504 +13
Misses 2594 2594
Flags with carried forward coverage won't be shown. Click here to find out more.
|
Pull Request Test Coverage Report for Build 6425912482
💛 - Coveralls |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Lgtm! Feel free to merge when the rebating is done and the tests pass.
595262d
to
3350539
Compare
Also fixes that input data types were inconsistent between dask arrays and computed numpy results.
3350539
to
9169abe
Compare
Reran my profiling and got the same results as before all this rebasing and refactoring so I think we're good here. |
As discussed in #2562, performance (memory usage for sure) is better when chunking is done based on resolution. Especially when native resampling is done. This means the 500m band's chunks are twice the size of the 1km band's chunks and 4 times the size of the 2km band's chunks. This should mean dask doesn't have to do any rechunking.
This PR is a continuation of #2052 and should be rebased/remerged after that is merged first. This PR also includes a fix for sunz correction where the dtype for 32-bit floats was being bumped up to 64-bit floats inside dask computations causing more memory usage than we thought.
AUTHORS.md
if not there already