Use primary context in CUDA GPU code. #3432
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Since LAMMPS uses the low-level driver API of CUDA, it needs to ensure that it is in the correct context when invoking such functions. At the moment it creates and switches to its own context inside `UCL_Device::set` but then assumes that the driver is still in that context for subsequent calls into CUDA; if another part of the program uses a different context (such as the CUDA runtime using the "primary" context) this will cause failures inside LAMMPS. This patch changes the context creation to instead use the primary context for the requested device. While it's not perfect, in that it still doesn't ensure that it's in the correct context before making driver API calls, it at least allows it to work with libraries that use the runtime API.
wmbrownIntel
reviewed
Sep 8, 2022
There was a problem hiding this comment.
@benmenadue I think it is OK to make this the default behavior, but please add preprocessor option for original behavior (e.g. #ifndef GERYON_UNIQUE_CONTEXT). Thanks!
@wmbrownIntel implemented your request |
akohlmey
approved these changes
Sep 9, 2022
wmbrownIntel
approved these changes
Sep 9, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This patch changes the CUDA context creation inside LAMMPS to instead use
the "primary" context for the requested device; this then makes LAMMPS
compatible with other libraries that also use CUDA via its runtime API.
It's not a complete fix, in that other libraries that use their own context
via the driver API could still break LAMMPS (e.g. if they switch to their
own context after LAMMPS has initialised its and don't switch it back before
returning), but it at least covers packages that use the CUDA runtime
library.
Unfortunately, to make all driver calls in LAMMPS "context aware" might
require more significant changes to the interfaces as the current context
may not be available at the calls (for example, the
_device_allocfunctionsin
nvd_memory.hthat don't take aUCL_Deviceargument).Related Issue(s)
Fixes: #3431
Author(s)
Ben Menadue, National Computational Infrastructure, Australian National University
Licensing
By submitting this pull request, I agree, that my contribution will be included in LAMMPS and redistributed under either the GNU General Public License version 2 (GPL v2) or the GNU Lesser General Public License version 2.1 (LGPL v2.1).
Backward Compatibility
These changes should not have any impact on backwards compatibility.
Implementation Notes
As recommended in the CUDA documentation, switches it to use
cuDevicePrimaryCtxRetainandcuDevicePrimaryCtxReleaseinstead ofcuCtxCreateandcuCtxDestroy.Post Submission Checklist