Use device arena for the_fa_arena when activating GPU-aware MPI #3362
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
WeiqunZhang
approved these changes
Jun 13, 2023
WeiqunZhang
added a commit
that referenced
this pull request
Jun 28, 2023
## Summary Implement a communications arena for comm buffers to replace `the_fa_arena`. It creates a separate arena when GPU-aware MPI is used and `the_arena` is not managed. ## Additional background The motivation for this is a communication performance degradation that is observed for GPU-aware MPI with `amrex.the_arena_is_managed=0`. @WeiqunZhang has a hypothesis that this may be due to the need for frequent re-registering of comm buffer pointers when using the same device arena as the other compute data. Hence a separate arena in this case would alleviate this issue. `the_fa_arena` is eliminated in this PR and the communication buffer directly uses `the_comms_arena` to simplify the code. ## Performance tests The above stated performance degradation is particularly observed with the `GPU/CNS/Exec/Sod` code under `Tests` and is alleviated by using a separate comms arena as seen in the performance data below. `original` refers to the state before we made the change in #3362 related to `the_fa_arena` pointing to the device arena which allowed `amrex.the_arena_is_managed=1` with GPU-aware MPI without a significant performance hit. It is compared with the current development branch and the proposed comms arena implementation. The data pointing to the performance improvement from this PR is highlighted.  In other tests such as the `ABecLaplacian` linear solve or the ERF code, using `amrex.the_arena_is_managed=0` did not show a significant performance hit and using this comms arena implementation did not harm the performance either. More comprehensive tests would be required to determine the effect on other codes and platforms. --------- Co-authored-by: Mukul Dave <mhdave@lbl.gov> Co-authored-by: Weiqun Zhang <WeiqunZhang@lbl.gov>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This change suggested by @WeiqunZhang points
the_fa_arenatoThe_Device_Arenawhen activating GPU-aware MPI. This obviates the need for settingthe_arena_is_managed=0to take advantage of GPU-aware MPI since it does not work well with managed memory.Additional background
The motivation for this PR is that this was an long-pending change but the immediate trigger was finding that GPU-aware MPI can reduce communication times significantly but that currently needs setting
the_arena_is_managed=0. Not setting this for GPU-aware MPI currently results in degraded performance.Past discussion on GPU-aware MPI: #2967
Preliminary performance test
Running 100 steps on 8 GPUs over 2 Perlmutter A100 nodes with
Tests/GPU/CNS/Exec/Sod,amr.n_cell = 128^3per GPU,amr.max_grid_size = 128,amrex.use_profiler_syncs = 1and setting optimal GPU affinities.Without
amrex.use_gpu_aware_mpi=1With
amrex.use_gpu_aware_mpi=1