-
Notifications
You must be signed in to change notification settings - Fork 859
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Q] srun + OpenMPI MPI_Comm_split_type #8299
Comments
Apologies for the delay here; we missed this issue as it was filed over the holidays. The issue was reported against the v4.0.x series, but I pre-emptively also applied the v4.1.x tag, too. |
@bosilca pointed me to the right place in the code. The problem here is that the split code requests the process locality for each proc participating in the split - this locality is then used for the split. OMPI's runtime provides that locality, but Slurm does not. At the moment, the code doesn't return an error if OMPI is unable to get the locality of each process - it just assumes that every proc is in its own region. PMIx v4 added support for computing locality to make it easier for RMs to implement it, but Slurm hasn't been updated yet to take advantage of it. That might be one solution, if someone wants to update Slurm to support the PMIx v4 features. |
During the discussion I proposed to throw an error if we are missing the information for the requested split type. I should have known better, one don't just raise errors in MPI, as we will automatically trigger the default error handler and abort the application. Thus, with the current MPI version there is little we can do, except returning a duplicate of MPI_COMM_SELF to put each process in its own communicator. |
Ah, good point! I had totally missed that one too. FWIW: it would be pretty simple to update Slurm to provide the required info, if someone is interested in doing so. Basically just one function call. |
NP. we use |
@artpol84 Since NVIDIA is the maintainer of the SLURM plugin, is there any chance you guys will add support for this? 😄 |
Slurm folks show no interest in supporting it, and no answer from NVIDIA |
I'm having trouble using
MPI_Comm_split_type
for in-node splits with customOMPI_COMM_TYPE_*
. Everything works fine when I run withmpirun
, but the same code doesn't work withsrun
. Is this supposed to work, or are there some limitations I'm not aware of? I'm trying with OpenMPI 4.0.3 and 4.0.5, Centos 7.7 with Slurm 19.05, stock hwloc 1.11 (but I also tried to compile OpenMPI with hwloc 2.4.0), pmix 3.1.5.This is a simple test app:
On our
EPYC 7742
system I get this withmpirun
:and this with
srun
Essentially, I get a split size 1 whatever type I use.
The text was updated successfully, but these errors were encountered: