Initialize low-level GPU API once instead of at each Variorum API call #443
Conversation
amarathe84
force-pushed
the
gpu-init-fix
branch
2 times, most recently
from
c897425
to
5150443
Compare
amarathe84
added
status-work-in-progress
amarathe84
added
status-ready-for-review
|
@amarathe84 This PR seems to be failing on lassen-gpu-only. Can you take a look? It could be an error with the runner and not the code. |
slabasan
force-pushed
the
gpu-init-fix
branch
from
0682a7c
to
a701bf0
Compare
slabasan
added
status-work-in-progress
|
During testing of this PR, we noticed double free corruption error for the Nvidia GPU build of Variorum on Lassen. Debugging this issue revealed that the double free corruption happens because the second variorum call made by the test application tries to use deallocated NVML device object without NVML being initialized. Resolution of this issue will require introduction of stateful variorum_init() and variorum_finalize() variorum APIs. The user of those APIs will need to manually call nvmlInit() after variorum_init() and nvmlShutdown() before variorum_finalize() in their code. This PR is in WIP state as it is waiting for the introduction of the advanced APIs in Variorum. |
|
#459 is tracking this for our gitlab tests. Closing this PR. |
Description
This PR adds state-preserving logic to the GPU functionality in Variorum. The goal is to call the respective initialization functions provided by the low-level device-specific APIs.
Fixes #427
Closes #459 once all tests pass.
Type of change
How Has This Been Tested?
Since this fix touches the AMD and Nvidia GPUs, it is tested on the following systems: