Oversubscription error appears when changing to 5.0.8 (from 4.1.4)

[skwde]

Background information

What version of Open MPI are you using? (e.g., v4.1.6, v5.0.1, git branch name and hash, etc.)

v5.0.8

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

5.0.8: micromamba using the conda-forge channel
4.1.4: spack

Please describe the system on which you are running

• Operating system/version: rocky 8
• Computer hardware:
• Network type: ib / high speed ethernet

---

Details of the problem

Please describe, in detail, the problem that you are having, including the behavior you expect to see, the actual behavior that you are seeing, steps to reproduce the problem, etc. It is most helpful if you can attach a small program that a developer can use to reproduce your problem.

Note: If you include verbatim output (or a code block), please use a GitHub Markdown code block like below:

In a slurm job I ask for 2 tasks and 8 cpus per task, making in total 16 cpus available.

While with version 4.1.4 the following worked without any issue:

NUMBA_NUM_THREADS=${SLURM_CPUS_PER_TASK:?}
mpiexec -n "${SLURM_NTASKS:?}" --map-by slot:pe="${NUMBA_NUM_THREADS:?}" python pi_hybrid.py

This throws an error with 5.0.8.

I don't get why, it seems MPI processes are also counted now? When I set NUMBA_NUM_THREADS to 7 it works but then 2 CPUS are basically unused because the MPI processes are idle during the numba parallelization.

Possible solutions are any of the following

mpiexec -n "${SLURM_NTASKS:?}" --cpus-per-proc "${NUMBA_NUM_THREADS:?}" python pi_hybrid.py
mpiexec -n "${SLURM_NTASKS:?}" --bind-to none python pi_hybrid.py
mpiexec -n "${SLURM_NTASKS:?}" --oversubscribe --map-by slot:pe="${NUMBA_NUM_THREADS:?}" python pi_hybrid.py

--cpus-per-proc seems the most obvious to me, but in the docs (https://docs.open-mpi.org/en/v5.0.8/man-openmpi/man1/mpirun.1.html#options-old-hard-coded-content-mdash-to-be-audited) the following is mentioend:

deprecated in favor of --map-by <obj>:PE=n

So I would rather use --map-by (which is only working when using --oversubscribe (no other <obj> seems to work, using core for instance just quits without giving any output in .out / .err) which seems counterintuitive.

Using --bind-to none is also not what I want because this does not ensure that all numba threads are on the same socket.

So what am I missing here?

[rhc54]

If you say this "throws an error", it would really be helpful if you told us "what error does it throw?" I assume from the rest of the text that it is something like "not enough slots", but that's just a guess.

Your command line should start two processes, each bound to 8 cpus. We do require that those 8 cpus all be on a single package - i.e., we will not split a process binding across packages. Not knowing how you configured Slurm, you might want to check that your allocation meets that requirement.

I have no idea what your "numba" is doing with those cpus and cannot help you there. I can take a quick peek to ensure that mpirun is doing its part.

[skwde]

Thanks for your response and sorry for not being specific enough...

--------------------------------------------------------------------------
Your job failed to map because the resulting process placement
would cause the process to be bound to CPUs in more than one
package:

  Mapping policy:  BYSLOT:NOOVERSUBSCRIBE
  Binding policy:  CORE:IF-SUPPORTED
  CPUs/rank:       8

This configuration almost always results in a loss of performance
that can significantly impact applications. Either alter the
mapping, binding, and/or cpus/rank policies so that each process
can fit into a single package, or consider using an alternative
mapper that can handle this configuration (e.g., the rankfile mapper).
--------------------------------------------------------------------------
�

I am not familiar with the term package. So I am not sure what you mean by that.

Though the way I requested resources, I get 16 CPUs.
To my understanding, slurm is aware of the sockets on the compute nodes and which cpus are on which socket.

So to my understanding I have 16 CPUs and I start 2 mpi processes each with 8 cpus.

numba is translating a python loop to machine code and multithreads similar to OpenMP.

[rhc54]

Community switched from using "socket" to "package" some time ago because boards now contain multi-chip packages, each chip containing lots of cpus and memory, that may or may not be attached to the motherboard via a socket. You can find explanation in the hwloc doc:

Package

    A processor Package is the physical package that usually gets inserted into a socket
on the motherboard. It is also often called a physical processor or a CPU even if these
names bring confusion with respect to cores and processing units. A processor package
usually contains multiple cores (and may also be composed of multiple dies). hwloc Package
objects were called Sockets up to hwloc 1.10. 

Best way to resolve something like that is to look at the topology. You might check the output of lstopo to see how many cpus/package you have. Is Slurm allocating at the cpu level as opposed to nodes? Generally problematic when running multi-cpu processes as the cpu's can be cherry-picked from all over the node, causing significant performance loss (though Slurm's allocator probably tries not to do that).

[skwde]

Thanks for the "definition" of package.

Slurm allocates based on CPUs (cores). Each node has two sockets. Each socket has 16 cores.

Can you confirm that my example is 2 MPI processes each with 8 CPUs should not require to oversubscribe when allocating 16 CPUs?

[rhc54]

No, it shouldn't - but that isn't the error you are reporting. The error being reported is that the allocation isn't contiguous on a package - i.e., you don't have 8 cpus on a single package for at least one of the processes.

So your Slurm is not configured to allocate entire nodes, but instead to allocate individual cpus - correct? Some places do that as a means of tightly packing allocations, though it is rare as it leads to allocations sharing nodes. Just trying to confirm the situation we are operating within.

I'd have to look into the code to see why "oversubscription" is having any impact on the behavior as it really shouldn't. It would also be nice to confirm that we aren't seeing some other problem and the error message is incorrectly being triggered.

Can you post your local topology - i.e., the xml file output from "lstopo" - so I can test it here?