There are not enough slots available

[sjackman]

Background information

Submitting a job to SLURM requesting 32 processes produces an error message. Adding the --oversubscribe option works around the issue. The system has 48 CPUs.

What version of Open MPI are you using?

OpenMPI 3.1.3

Describe how Open MPI was installed

OpenMPI was installed using Linuxbrew. brew install open-mpi https://linuxbrew.sh

Please describe the system on which you are running

• Operating system/version: CentOS 6.7
• Computer hardware: Intel Xeon E5-2650
• Network type: Ethernet

---

Details of the problem

Requesting 32 processes fails.

sbatch -c32 mpirun -np 32 …

There are not enough slots available in the system to satisfy the 32 slots
that were requested by the application:
  ABYSS-P

Either request fewer slots for your application, or make more slots available
for use.

Adding --oversubscribe works around the issue.

sbatch -c32 mpirun --oversubscribe -np 32 …

works as expected.

The system has 48 CPUs.

grep -c processor /proc/cpuinfo
48

[bgoglin]

I guess this is E5-2650v2, which has 12 cores per package, hence 24 cores total and 48 hyperthreads. But hyperthreads are not used by default, so only 24 slots are available here.

[jsquyres]

Even if hyperthreading is enabled (Linux is assumedly showing you the 48 hyperthreads), Open MPI defaults to one process per core, not one process per hyperthread. Hence, @bgoglin is right: if you only have 24 actual cores, then Open MPI will complain (by default) if you try to run more than 24 processes.

[sjackman]

Okay. That explains that. SLURM is configured to allow 48 tasks per node. Is there a way to tell OpenMPI to allow one process per thread rather than one per core?

[jsquyres]

Yes, use the --use-hwthread-cpus option to mpirun. This tells Open MPI's mapper to use HW threads as the mapping unit, not cores.

[sjackman]

Thanks, Jeff! When the requested number of slots is larger than the number of CPUs but less than or equal to the number of hardware threads, I'd suggest that --use-hwthread-cpus should be the default behaviour. Failing that, the error message could suggest that the user enable --use-hwthread-cpus.

[jsquyres]

We've had many a debate over this topic over the years. 😄

FWIW: using Intel hyperthreads still invokes a bit of a performance penalty, so it's not the default / we bias towards scheduling on cores. And we make it a little hard to use them (e.g., --use-hwthread-cpus)

[sjackman]

Seems reasonable if that's an intentional design decision. I do think the error message could be improved though to suggest --use-hwthread-cpus.

[jsquyres]

Fair enough: I filed #6061 to improve the help message.

It's on tap to go into the v4.0.x release, but probably not until 4.0.1. I filed PR's for the 3.0.x and 3.1.x branches too; we'll see if the RMs allow it for those release trains.

[jsquyres]

Good to close this one out?

[sjackman]

Thank you, Jeff! You go above and beyond.

3. Resource manager (e.g., SLURM, PBS/Torque, LSF, etc.)

I'm running this job within SLURM. Should it have received the number of available slots from SLURM?

[jsquyres]

Yes. It depends on how SLURM was configured as to whether it gave you a number of cores or a number of hyperthreads (IIRC, it defaults to cores).

What happens if you:

$ cat myscript.sh
#!/bin/sh
grep -c processor /proc/cpuinfo | wc -l
mpirun -np 32 …
$ sbatch -c32 myscript.sh

[sjackman]

Yes, that is how I launched the job. sbatch -c32 mpirun -np 32 …, which caused the error the error that I'm seeing here. The simple solutions are to either use
batch -c24 mpirun -np 24
or
sbatch -c32 mpirun -np 32 --use-hwthread-cpus …
I don't see an obvious way of requesting 32 threads rather than 32 cores. I wonder if perhaps
sbatch --cpus-per-task=16 --threads-per-core=2 mpirun -np 32 …
would work.

You can close this ticket now though. Thanks for all your help!