-
Notifications
You must be signed in to change notification settings - Fork 856
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
MPI collective writes causing MPI_Finalize to segfault in mca_btl_vader_endpoint_xpmem_rcache_cleanup() #6524
Comments
@cniethammer how many nodes are you running on ? |
The test was run on a single node. |
free the component mpool in mca_btl_vader_component_close() and after freeing soem objects that depend on it such as mca_btl_vader_component.vader_frags_user Thanks Christoph Niethammer for reporting this. Refs. open-mpi#6524 Signed-off-by: Gilles Gouaillardet <gilles@rist.or.jp>
@cniethammer can you please manually download and apply the patch at https://github.com/open-mpi/ompi/pull/6526.patch ? |
Unfortunately the patch did not solve the problem. |
@cniethammer I found an other issue that is specific to meanwhile, disabling xpmem should do the trick if |
Strike! Changing the vader single copy mechanism seems to solve the issue. |
Refs. open-mpi#6524 Signed-off-by: Gilles Gouaillardet <gilles@rist.or.jp>
free the component mpool in mca_btl_vader_component_close() and after freeing soem objects that depend on it such as mca_btl_vader_component.vader_frags_user Thanks Christoph Niethammer for reporting this. Refs. open-mpi#6524 Signed-off-by: Gilles Gouaillardet <gilles@rist.or.jp> (cherry picked from commit open-mpi/ompi@77060ca)
The fix is scheduled for release in OpenMPI 4.0.2. Upstream issue: open-mpi/ompi#6524
This commit fixes an issue discovered in the XPMEM registration cache. It was possible for a registration to be invalidated by multiple threads leading to a double-free situation or re-use of an invalidated registration. This commit fixes the issue by setting the INVALID flag on a registation when it will be deleted. The flag is set while iterating over the tree to take advantage of the fact that a registration can not be removed from the VMA tree by a thread while another thread is traversing the VMA tree. References open-mpi#6524 References open-mpi#7030 Closes open-mpi#6534 Signed-off-by: Nathan Hjelm <hjelmn@google.com>
This commit fixes an issue discovered in the XPMEM registration cache. It was possible for a registration to be invalidated by multiple threads leading to a double-free situation or re-use of an invalidated registration. This commit fixes the issue by setting the INVALID flag on a registation when it will be deleted. The flag is set while iterating over the tree to take advantage of the fact that a registration can not be removed from the VMA tree by a thread while another thread is traversing the VMA tree. References open-mpi#6524 References open-mpi#7030 Closes open-mpi#6534 Signed-off-by: Nathan Hjelm <hjelmn@google.com>
This commit fixes an issue discovered in the XPMEM registration cache. It was possible for a registration to be invalidated by multiple threads leading to a double-free situation or re-use of an invalidated registration. This commit fixes the issue by setting the INVALID flag on a registation when it will be deleted. The flag is set while iterating over the tree to take advantage of the fact that a registration can not be removed from the VMA tree by a thread while another thread is traversing the VMA tree. References open-mpi#6524 References open-mpi#7030 Closes open-mpi#6534 Signed-off-by: Nathan Hjelm <hjelmn@google.com> (cherry picked from commit f86f805)
This commit fixes an issue discovered in the XPMEM registration cache. It was possible for a registration to be invalidated by multiple threads leading to a double-free situation or re-use of an invalidated registration. This commit fixes the issue by setting the INVALID flag on a registation when it will be deleted. The flag is set while iterating over the tree to take advantage of the fact that a registration can not be removed from the VMA tree by a thread while another thread is traversing the VMA tree. References open-mpi#6524 References open-mpi#7030 Closes open-mpi#6534 Signed-off-by: Nathan Hjelm <hjelmn@google.com> (cherry picked from commit f86f805)
This problem appears to be resolved in v4.0.x at 803bdd6 on a Cray XC running CLE 7.0UP02 and xpmem 2.2.20. |
Background information
What version of Open MPI are you using? (e.g., v1.10.3, v2.1.0, git branch name and hash, etc.)
openmpi-4.0.1rc3
Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)
Installation from tarball with
Please describe the system on which you are running
Cray XC40, PrgEnv-gnu/6.0.4
gcc 8.2.0
xpmem/2.2.4-6
Details of the problem
Using MPI collective writes (MPI_File_write_all, MPI_File_write_at_all) on NFS shared file system causes MPI_Finalize() to segfault for a total write of >= 6MB of data:
A debug build with --enable-debug reports
mpirun -n 24 ./test .../openmpi-4.0.1rc3/opal/mca/btl/vader/btl_vader_xpmem.c:160: mca_btl_vader_endpoint_xpmem_rcache_cleanup: Assertion `((0xdeafbeedULL << 32) + 0xdeafbeedULL) == ((opal_object_t *) (reg))->obj_magic_id' failed.
Note: The same test on a Lustre file system works fine.
The test program used is:
The text was updated successfully, but these errors were encountered: