MPI collective writes causing MPI_Finalize to segfault in mca_btl_vader_endpoint_xpmem_rcache_cleanup()

[cniethammer]

Background information

What version of Open MPI are you using? (e.g., v1.10.3, v2.1.0, git branch name and hash, etc.)

openmpi-4.0.1rc3

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

Installation from tarball with

./configure --enable-shared --enable-static --enable-mpi-thread-multiple

Please describe the system on which you are running

Cray XC40, PrgEnv-gnu/6.0.4
gcc 8.2.0
xpmem/2.2.4-6

---

Details of the problem

Using MPI collective writes (MPI_File_write_all, MPI_File_write_at_all) on NFS shared file system causes MPI_Finalize() to segfault for a total write of >= 6MB of data:

ranks	fails	works
24	256*1024	255*1024
12	512*1024	511*1024

A debug build with --enable-debug reports

mpirun -n 24 ./test
.../openmpi-4.0.1rc3/opal/mca/btl/vader/btl_vader_xpmem.c:160: mca_btl_vader_endpoint_xpmem_rcache_cleanup: Assertion `((0xdeafbeedULL << 32) + 0xdeafbeedULL) == ((opal_object_t *) (reg))->obj_magic_id' failed.

Note: The same test on a Lustre file system works fine.

The test program used is:

#include "mpi.h"
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

#include "mpi.h"

int main(int argc, char *argv[]) {
  int rank, size;
  MPI_Init(&argc, &argv);
  MPI_Comm_size(MPI_COMM_WORLD, &size);
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
  
  int *buffer;
  long BUFFER_SIZE = (512 * 1024); /* # elements in buffer *for each MPI proc */
  if (argc > 1) {
    BUFFER_SIZE = atol(argv[1]);
  }
  int i;
  buffer = (int *)malloc(BUFFER_SIZE * sizeof(int));
  for (i = 0; i < BUFFER_SIZE; i++) {
    buffer[i] = rank * BUFFER_SIZE + i;
  }

  MPI_File fh;
  MPI_File_open(MPI_COMM_WORLD, "data.bin", MPI_MODE_CREATE | MPI_MODE_RDWR,
                MPI_INFO_NULL, &fh);

  int count = 1;
  int blocklength = BUFFER_SIZE;
  int stride = BUFFER_SIZE;
  MPI_Datatype etype = MPI_INT;
  MPI_Datatype filetype;
  MPI_Type_vector(count, blocklength, stride, etype, &filetype);
  MPI_Type_commit(&filetype);

  MPI_Status status;
/* Both methods end in the same error; Changing to non collective versions works
 * fine. */
#if 0
    MPI_Offset disp = (MPI_Offset)rank * BUFFER_SIZE * sizeof(int);
    MPI_File_set_view(fh, disp, etype, filetype, "native", MPI_INFO_NULL);
    MPI_File_write_all(fh, (void *)buffer, BUFFER_SIZE, etype, &status);
#else
  MPI_File_write_at_all(fh, BUFFER_SIZE * rank, (void *)buffer, BUFFER_SIZE,
                        etype, &status);
#endif

  MPI_File_close(&fh);
  free(buffer);

  MPI_Finalize();

  return 0;
}

[ggouaillardet]

@cniethammer how many nodes are you running on ?

[cniethammer]

The test was run on a single node.

[ggouaillardet]

@cniethammer can you please manually download and apply the patch at https://github.com/open-mpi/ompi/pull/6526.patch ?

[cniethammer]

Unfortunately the patch did not solve the problem.

[ggouaillardet]

@cniethammer I found an other issue that is specific to xpmem (double free issue)
unfortunately, I do not know when I will find the time to investigate it.

meanwhile, disabling xpmem should do the trick
mpirun --mca btl_vader_single_copy_mechanism cma ...

if cma is not available on cray, then you can
mpirun --mca btl_vader_single_copy_mechanism none ...
(and that will very likely impact performances)

[cniethammer]

Strike! Changing the vader single copy mechanism seems to solve the issue.

[hppritcha]

This problem appears to be resolved in v4.0.x at 803bdd6 on a Cray XC running CLE 7.0UP02 and xpmem 2.2.20.