Improved station sampler #2838
Merged
Improved station sampler #2838
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…_stataion_sampler' into feature/ygyu/revise_stataion_sampler
metdyn
added
the
0 Diff Trivial
tclune
reviewed
May 21, 2024
tclune
reviewed
May 21, 2024
tclune
reviewed
May 21, 2024
tclune
reviewed
May 21, 2024
tclune
reviewed
May 21, 2024
tclune
reviewed
May 21, 2024
tclune
reviewed
May 21, 2024
tclune
reviewed
May 21, 2024
|
Thank you, Tom. I need to do a few control experiment to examine the effect on netCDF write and how much regridding time has changed. I am closing this PR to work offline. |
- removed chunksizes in station sampler
|
work offline for improvement |
Major efficiency issues have been resolved. |
metdyn
dismissed
tclune’s stale review
The code has been changed significantly since the last review.
tclune
approved these changes
Jun 24, 2024
🤯 |
|
Something is wrong with CircleCI. It's not reporting the success. Merging with powers. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Types of change(s)
Checklist
make tests)Description
The previous version for station sampler which is the first (and the simplest) sampler we implemented a year ago shows significant slowness for NOAA GHCNd (124954 stations). The previous version used LocStream on root only. Now we have added intermediate steps: LS(uniformly distributed on cores), LS (w CS bkg), MPI_gatherV. Running ExtDatadirver.x on discover with 8640 cores for C1440-L137 shows 1.39 s for GHCNd for 1 write statement with 10 3d-variables.
Summary: 490 s runtime is reduced to 6.9 s for station sampler with GHCNd input