MCA settings required to run a windowing query

[mathstuf]

Background information

What version of Open MPI are you using? (e.g., v3.0.5, v4.0.2, git branch name and hash, etc.)

v4.0.2

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

Fedora 31

Please describe the system on which you are running

• Operating system/version: Fedora 31
• Computer hardware: Intel i7
• Network type: Local?

---

Details of the problem

Probably related to #6275 (which is where I got the OMPI_MCA_osc=sm,pt2pt suggestion). I'm not completely sure what exactly is going wrong here (came across this in a test suite of a project I'm helping out with for CI). Reproducer with command line below:

#include <vector>
#include <cassert>
#include <mpi.h>

int main(int argc, char** argv)
{
    MPI_Init(&argc, &argv);

    int rank, size;
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);

    int width = 4;

    using T = int;
    std::vector<T> buffer_(width);
    MPI_Win window_;
    MPI_Win_create(buffer_.data(), buffer_.size()*sizeof(T), sizeof(T), MPI_INFO_NULL, MPI_COMM_WORLD, &window_);

    MPI_Win_lock_all(MPI_MODE_NOCHECK, window_);

    int target_rank = (rank + 2) % size;
    std::vector<int> out(width);
    for (int i = 0; i < width; ++i)
    {
        out[i] = target_rank * width + i;
        MPI_Put(&out[i], 1, MPI_INT, target_rank, i, 1, MPI_INT, window_);
    }
    MPI_Win_flush(target_rank, window_);

    MPI_Barrier(MPI_COMM_WORLD);

    int source_rank = (rank + 1) % size;
    std::vector<int> values(width);
    for (int i = 0; i < width; ++i)
    {
        MPI_Get(&values[i], 1, MPI_INT, source_rank, i, 1, MPI_INT, window_);
    }
    MPI_Win_flush_local(source_rank, window_);

    for (int i = 0; i < width; ++i)
    {
        assert(values[i] == source_rank*width + i);
    }

    MPI_Win_unlock_all(window_);

    MPI_Finalize();

    return 0;
}

Compile line (adapted and minimized from a CMake-generated command line):

/usr/bin/g++ -DOMPI_SKIP_MPICXX -isystem /usr/include/openmpi-x86_64 -pthread -o mpi-window-standalone -Wl,-rpath -Wl,/usr/lib64/openmpi/lib /usr/lib64/openmpi/lib/libmpi.so ../mpi-window-standalone.cpp

Either of these works:

% ./mpi-window-standalone
% OMPI_MCA_osc=sm,pt2pt /usr/lib64/openmpi/bin/mpiexec ./mpi-window-standalone

This fails:

% /usr/lib64/openmpi/bin/mpiexec ./mpi-window-standalone
[myhostname:1079189] Wrote -1, expected 4, errno = 14
[myhostname:1079189] Wrote -1, expected 4, errno = 14
[myhostname:1079189] Wrote -1, expected 4, errno = 14
[myhostname:1079189] Wrote -1, expected 4, errno = 14
[myhostname:1079189] Wrote -1, expected 4, errno = 14
<snip…and so on>

Cc: @mrzv

[hjelmn]

I do not recommend setting the osc MCA variable to sm,pt2pt. The pt2pt component is slow and will likely go away in a future release.

Also, MPI_Win_create() is discouraged in favor of MPI_Win_allocate(). The later call gives the MPI implementation control over the buffer which may give better performance (and will probably mask your CMA issue).

The problem appears to be a failure in CMA. Looks like you are running with your MPI processes in different containers. This is a well understood issue and you need to either enable ptrace for your docker containers or run all of your MPI processes in the same container.

[mathstuf]

Seems I missed adding that this also happens on a machine directly (without the container). And I don't know how to set up MPI processes to run in different containers than the one I launch it in. Running with --privileged didn't remove the error either.

[mathstuf]

Indeed, using MPI_Win_allocate seems to prevent the error. Is that the recommended fix or should the issue with MPI_Win_create still be investigated?

[hjelmn]

The CMA error is worth investigating since it is happening outside a container. I can take a look.

[hjelmn]

And, yes. Please use allocate instead of create.

[mathstuf]

I can confirm that using allocate instead of create in the actual test case avoided the problem. Thanks!

[mathstuf]

So it seems that the signature and/or usage of MPI_Win_allocate is confusing (at least to me).

int MPI_Win_allocate(MPI_Aint size, int disp_unit, MPI_Info info, 
                  MPI_Comm comm, void *baseptr, MPI_Win *win)

and then baseptr is documented as an output parameter? Is it not being void** a mistake then (not that it can be fixed anymore)? Is there really a *(void**)baseptr = newmem; inside there somewhere?

It seems that MPI_Win_allocate cannot be used for this case because it wants to allocate the memory whereas with MPI_Win_create, the memory to use is controlled by a backing vector we can pass in.

[devreal]

@mathstuf This is not a mistake but by design (see MPI v3.1, Section 2.5.5 Rationale in Section 8.2). The rational is that C allows you to cast any pointer type to void * (the generic pointer) but you cannot pass int** for a void** parameter because it would require a pointer to a generic pointer, losing its generic character.

Example (i is of type int*):

$ gcc test_ptr.c 
test_ptr.c: In function ‘main’:
test_ptr.c:11:8: warning: passing argument 1 of ‘test’ from incompatible pointer type [-Wincompatible-pointer-types]
   test(&i);
        ^
test_ptr.c:2:6: note: expected ‘void **’ but argument is of type ‘int **’
 void test(void **a)
      ^~~~

I tested your use-case with current master and it works on my local machine. I cannot really make sense of the output you posted here though.

[mathstuf]

Hmm. I guess I'm getting rusty on my C type system and want it to be more like Rust :) . I'll take a closer look at it.