Halos - making better use of halos to improve parallel performance #220
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This breaks the tests (so I am knowingly commiting a broken state), but I want to show that it is the halo regions that are responsible.
Blessed output excludes halo region from the calculations.
Extend horizontal loops to make the model compute some variables inside the halos. This will remove the need for the extra exchange calls. Also, fixes #219
Partially fixes #218
Update initial blessed eta field for outcropping test
Codecov Report
@@ Coverage Diff @@
## master #220 +/- ##
==========================================
- Coverage 91.82% 91.76% -0.06%
==========================================
Files 20 20
Lines 1822 1810 -12
Branches 96 95 -1
==========================================
- Hits 1673 1661 -12
Misses 124 124
Partials 25 25
Continue to review full report at Codecov.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As noted in #218, there are a lot of exchange calls that could be removed if the halos were being used more efficiently. This PR does just that, and partially closes #218.
There are a few bugs fixed in this PR:
eta_starvalues from the halo that had never been computed. Oops. It's fixed by wrappingeta_starin that subroutine. The performance hit is irrelevant, since the internal solver is orders of magnitude slower, and can only be used on a single core.The real meat of this PR is flushing many exchange calls, which leads to substantial improvements in scaling and performance improvement when running in parallel. The 1.5 layer version is now faster when running on multiple cores than it is on a single core, which is good. The previous scaling was embarrassing.
Still to do: