v4.x: Using --map-by core:span causes error in MPI_Init

[lyu]

Background information

What version of Open MPI are you using?

4.0.4 from release tarball. Can't test the git version since it requires pandoc.

Describe how Open MPI was installed

OpenMPI configuration:
./configure --prefix=$PREFIX --without-verbs --with-hwloc=internal

Please describe the system on which you are running

• Operating system/version: CentOS Linux release 7.5.1804
• Computer hardware: Intel Xeon CPU E5-2690 v3
• Network type: Mellanox Technologies MT27600 [Connect-IB]

A few more systems were tested (laptops, ARM machines) and they all have the same issue.

---

Details of the problem

Compile the following MPI program:

#include "mpi.h"

int main() {
    MPI_Init(NULL, NULL);
    MPI_Finalize();
}

Run it with
mpirun -n 4 --map-by core:span ./a.out

Error message:

[login2:55588] PMIX ERROR: NOT-FOUND in file dstore_base.c at line 2866
[login2:55588] PMIX ERROR: NOT-FOUND in file server/pmix_server.c at line 3408
[login2:55592] PMIX ERROR: OUT-OF-RESOURCE in file client/pmix_client.c at line 231
[login2:55592] OPAL ERROR: Error in file pmix3x_client.c at line 112
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[login2:55592] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[21660,1],0]
  Exit code:    1
--------------------------------------------------------------------------
[login2:55588] PMIX ERROR: ERROR in file gds_ds21_lock_pthread.c at line 99
[login2:55588] PMIX ERROR: ERROR in file gds_ds21_lock_pthread.c at line 99

I tested this on both the login node and the compute node, the error persists as long as I pass --map-by core:span or --map-by numa:span to mpirun.

OpenMPI 4.0.3 works fine.

[tonycurtis]

BTW you should be able to just create a shell script on your PATH that does exit 0 or equivalent called pandoc ( or symlink to /bin/true for example). That will pass. (Required by PMIx if memory serves.)

[hppritcha]

I'm able to reproduce this.

[jsquyres]

BTW you should be able to just create a shell script on your PATH that does exit 0 or equivalent called pandoc ( or symlink to /bin/true for example). That will pass. (Required by PMIx if memory serves.)

Or you could just download a pandoc binary from https://github.com/jgm/pandoc/releases/ 😄

[lyu]

OK, built OMPI from the master branch, the error messages are:

$ mpirun -n 4 --map-by core:span ./a.out 
--------------------------------------------------------------------------
A request for multiple cpus-per-proc was given, but a directive
was also given to map to an object level that has less cpus than
requested ones:

  #cpus-per-proc:  1
  number of cpus:  0
  map-by:          UNKNOWN

Please specify a mapping level that has more cpus, or else let us
define a default mapping that will allow multiple cpus-per-proc.
--------------------------------------------------------------------------

Not sure what happened, but looks like a different issue, or simply because of changes to launch parameters that I am unaware of.

[jsquyres]

Yes, master and the v4.x branches have an entirely different back-end for run-time support (i.e., master is what will eventually become v5.0.0, and that has had a complete revamp of the run-time system compared to the v4.0.x/v4.1.x series).

[hppritcha]

@jsquyres other binding options also exhibit this behavior

[jsquyres]

@hppritcha Mapping or binding or both? Can you provide a list?

[hppritcha]

I'm busy bisecting

[hppritcha]

git bisect is implicating 7643 commit 79c4268