Cannot specify PE (processing elements) in `PRTE_MCA_rmaps_default_mapping_policy`

[satishskamath]

Thank you for taking the time to submit an issue!

Background information

What version of Open MPI are you using? (e.g., v4.1.6, v5.0.1, git branch name and hash, etc.)

  1) 2025                           5) GCC/14.2.0                      9) libpciaccess/0.18.1-GCCcore-14.2.0  13) UCX/1.18.0-GCCcore-14.2.0       17) UCC/1.3.0-GCCcore-14.2.0
  2) GCCcore/14.2.0                 6) numactl/2.0.19-GCCcore-14.2.0  10) hwloc/2.11.2-GCCcore-14.2.0         14) libfabric/2.0.0-GCCcore-14.2.0  18) OpenMPI/5.0.7-GCC-14.2.0
  3) zlib/1.3.1-GCCcore-14.2.0      7) XZ/5.6.3-GCCcore-14.2.0        11) OpenSSL/3                           15) PMIx/5.0.6-GCCcore-14.2.0
  4) binutils/2.42-GCCcore-14.2.0   8) libxml2/2.13.4-GCCcore-14.2.0  12) libevent/2.1.12-GCCcore-14.2.0      16) PRRTE/3.0.8-GCCcore-14.2.0

v5.0.7

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

Easybuild.

Please describe the system on which you are running

• Operating system/version: Red Hat Enterprise Linux release 9.4 (Plow)
• Computer hardware: AMD rome/genoa
• Network type: Infiniband

---

Details of the problem

[satishk@tcn3 hybrid_mpi_openmp]$ mpirun -np 2 --report-bindings --prtemca rmaps_default_mapping_policy "slot:PE=2"  ./hybrid_new 
[satishk@tcn3 hybrid_mpi_openmp]$ mpirun -np 2 --report-bindings --prtemca rmaps_default_mapping_policy "slot"  ./hybrid_new 
[tcn3.local.snellius.surf.nl:2467403] Rank 0 bound to package[0][core:0]
[tcn3.local.snellius.surf.nl:2467403] Rank 1 bound to package[0][core:1]
Hello from thread 0 out of 2 from process 1 out of 2 on tcn3.local.snellius.surf.nl
Hello from thread 1 out of 2 from process 1 out of 2 on tcn3.local.snellius.surf.nl
Hello from thread 0 out of 2 from process 0 out of 2 on tcn3.local.snellius.surf.nl
Hello from thread 1 out of 2 from process 0 out of 2 on tcn3.local.snellius.surf.nl

PRRTE environment variables are not working as expected.
Based on the documentation of prte_info, I should be able to specify processing elements within slots but it does not produce any output.
Documentation:

               MCA rmaps: "rmaps_default_mapping_policy" (current value: "", data source: default, type: string)
                          Default mapping Policy [slot | hwthread | core | l1cache | l2cache | l3cache | numa | package | node | seq | dist | ppr | rankfile | pe-list=a,b
                          (comma-delimited ranges of cpus to use for this job)], with supported colon-delimited modifiers: PE=y (for multiple cpus/proc), SPAN, OVERSUBSCRIBE,
                          NOOVERSUBSCRIBE, NOLOCAL, HWTCPUS, CORECPUS, DEVICE=dev (for dist policy), INHERIT, NOINHERIT, ORDERED, FILE=%s (path to file containing sequential or
                          rankfile entries)

[rhc54]

I see two cmd lines, but only one set of output. Are you saying the first cmd line output nothing, and the second cmd line generated four processes??

[satishskamath]

Hi @rhc54 ,
Indeed, I am testing a hybrid MPI/OpenMP program so there are two ranks and each rank gets two threads so totally 4 outputs.
The first cmd line does not generate any output at all because I am specifying PE along with the slot.

Even simpler:

[satishk@tcn3 hybrid_mpi_openmp]$ mpirun -np 2  --prtemca rmaps_default_mapping_policy slot  hostname
tcn3.local.snellius.surf.nl
tcn3.local.snellius.surf.nl
[satishk@tcn3 hybrid_mpi_openmp]$ mpirun -np 2  --prtemca rmaps_default_mapping_policy slot:PE=2  hostname
[satishk@tcn3 hybrid_mpi_openmp]$ 

[rhc54]

I see - well, fwiw, the output from the 2nd cmd line looks correct. I'll have to take a look at the first cmd line, but I suspect the issue is just stale versions - it probably works fine with newer versions. If you want to try, use the release candidates for PMIx v5.0.10 and PRRTE v3.0.13:

https://github.com/openpmix/openpmix/releases/tag/v5.0.10rc1
https://github.com/openpmix/prrte/releases/tag/v3.0.13rc1

[rhc54]

I checked and we actually don't currently support including pes/rank in the default policy. Not entirely clear to me why I originally did that, so I'm correcting it (see openpmix/prrte#2335). Will be included in the next OMPI v5 update.

For whatever reason, the error message informing you of that situation wasn't being output by your first cmd line. That much was at least fixed in newer PRRTE releases. Still, support for including pes/rank in the default policy wasn't there...but will be soon.

[satishskamath]

Thanks Ralph. Is there any other environment variable via which we can set PES/rank in the current OMPI releases? Because we can definitely set it using --map-by option on the command line. Was looking for an equivalent environment variable.

[rhc54]

Afraid not - only place was in the MCA param you were using, and I only added the pe support just last week. You can, however, download the following tarballs and use them (added .txt to the end so GitHub would post them) as they include the update and will be in OMPI v5.0.10:

pmix-5.0.10rc2.tar.bz2.txt

prrte-3.0.13rc2.tar.bz2.txt

[casparvl]

Because we can definitely set it using --map-by option on the command line. Was looking for an equivalent environment variable.

Just to clarify for those that might come after us, looking for a solution: what @satishskamath meant by this is that

$ mpirun -np 2 --map-by slot:PE=2 --report-bindings hostname
[tcn440.local.snellius.surf.nl:2125905] Rank 0 bound to package[1][core:64-65]
[tcn440.local.snellius.surf.nl:2125905] Rank 1 bound to package[1][core:66-67]
tcn440.local.snellius.surf.nl
tcn440.local.snellius.surf.nl

still works as expected, you just cannot achieve the same through environment variables (which, for our workflow, would have been easier in practice).

Anyway, thanks for your prompt responses and fix @rhc54 , much appreciated!

[rhc54]

Glad it could be done!