-
Notifications
You must be signed in to change notification settings - Fork 29
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unified virtual memory failure in multi-GPU box #41
Comments
Feedback from NVIDIA: They said: "The documentation for cuPointerGetAttribute is here When testing for the CU_POINTER_ATTRIBUTE_MEMORY_TYPE, the possible values for data are CU_MEMORYTYPE_HOST and CU_MEMORYTYPE_DEVICE only. The other two options are never returned by this call. A different call to cuPointerGetAttribute is needed to determine if a pointer points to managed memory. Testing for the CU_POINTER_ATTRIBUTE_IS_MANAGED, sets data to either 0 or 1. " Thus, I think they say it looks like psm_user.h _psmi_is_cuda_mem is being used to indicate the CUDA IPC can be used, but is false triggering when the memory is managed, according to the CUDA documentation. However, I disagree with them: I have read the documentation, and it is actually unclear what the behaviour is defined as under unified memory pointers. In fact it seems to miss a few words to say what pointer type is being talked about in places. "Returns in *data the physical memory type of the memory that ptr addresses as a CUmemorytype enumerated value." Can reasonably be interpreted that the Unified memory type should return CU_MEMORYTYPE_UNIFIED, on account of it being unified memory type. It seems very very odd to have CU_MEMORYTYPE_UNIFIED and a query function for MEMORY_TYPE that is unable to return some of the values. Whatever happens: the combination of CUDA and PSM2 has a bug until this is fixed and the usage agreed upon. |
Following appears to fix the issue (psm_user.h).
Personally, I think cuPointerGetAttribute looks to be in the wrong, though |
The cuda 10.1 execution of the code was run on a system with the cuda 9.2 kernel driver. it is possible (Tim Lanfear produced same output with cuda 9.2 and 10.1) that the 10.1 behaviour will match 9.2 once the kernel driver is updated. This is on a centrally run supercomputer, so I can't update kernel drivers to check. |
Received the following from Nvidia, and they confirm this is their defined behaviour of cuPointerGetAttribute. Perhaps even worse, I think this means the returned class will vary page by page with UVM according to location:
|
Reviewing old issues - it appears that Adam submitted a pair of patches for this back in 2019. @paboyle - I know it's been a ridiculously long time but do you know if this was fixed in more recent IFS releases or is this still a problem for you? |
Pete has agreed that PSM2 was patched to correct the issue. |
See:
open-mpi/ompi#6799
I'm running OpenMPI 4.0.1 self compiled over Omnipath with IFS 10.8, as distributed by Intel.
The boards are
HPE XA with
4 x Nvidia Volta V100 GPU's and
4 OPA 100Gb ports on two PCIe dual port HFI cards.
The good news is that MPI appears to work between nodes, where these buffers are sent from explicit device memory.
However when I run four MPI ranks per node and ensure that communications between ranks use unified virtual memory (UVM) allocated with cudaMallocManaged(), I get a failure:
When I run with a patch to the code to use explicit host memory the code succeeds.
However, I want to be able to run these buffers from UVM and have loops with either host or device execution policy fill them, as that is how the code was designed to operate.
In the meantime tracked all the way back through PSM2 to a bug in CUDA (recompiled PSM2 from 10.8 released source, and inserted printf debugging, had a lovely 24h...)
Source:
Edited psm_user.h and _psmi_is_cuda_mem to instrument.
Appears that a bug in cuPointerGetAttribute causes PSM2 to interpret UVM
pointers as Device pointers, and try to use Cuda IPC to communicate
intra-node, with subsequent failure.
A simple 16 line of code example that fails:
———————
CUDA is supposed to returning one of:
But it is instead reporting that Unified memory is device memory incorrectly, causing PSM2 to do bad things, like think it can use Cuda IPC and then failing as above.
——————
CUDA 9.2
——————
nvcc simple.cc -o simple.x -lcuda
This causes the test in PSM2 to interpret UVM pointers as CU_MEMORYTYPE_DEVICE and try and fail to use Cuda IPC on UVM.
——————
CUDA 9.1 .. Officially supported with IFS 10.8
——————
——————
CUDA 10.1
——————
Behaves differently, returns error, but still does not produce
CU_MEMORYTYPE_UNIFIED => Bug in current version of CUDA.
However, absence of cudaSuccess will probably make PSM2 work. Except on our supercomputer system the kernel driver in IFS 10.8 does not support CUDA 10.1, failing with:
CUDA driver version is insufficient for CUDA runtime version
Will report this to CUDA, and get the kernel driver updated, which requires persuading a reluctant vendor that this is a genuine support need.
Even the CUDA 10.1 version does not return CU_MEMORYTYPE_UNIFIED and returns invalid arguments error, so it is still buggy, but my reading of the PSM2 source is that this will now be a benign bug since the absences of cudaSuccess return will stop PSM2 being tricked by CUDA.
The text was updated successfully, but these errors were encountered: