PMIx_Abort not always aborting

[jsquyres]

@rhc54 and I have talked about this on the phone and IM. There are cases where PMIx_Abort will actually deadlock instead of completing aborting. Ralph knows what the problem is; he hasn't figured out how to fix it yet. Filing this issue to track it.

This is causing some MPI and OSHMEM tests to hang on the Cisco MTT cluster, and eat up lots of CPU.

I'm seeing this on master and 2.x (I haven't started testing 3.x yet, but I'm sure the problem is over there, too).

[jsquyres]

Unfortunately, it does not look like this was solved.

In the https://www.open-mpi.org/nightly/master/openmpi-master-201703230209-ea84a53.tar.bz2 tarball, it included 55e4fba which fixed one of the code paths. However, I see a Cisco MTT process from that tarball that is still hung this morning. I see MPI_Errhandler_fatal_f and MPI_Errhandler_fatal_c processes hung.

This might be a different code path than the OSHMEM failures, though -- I cannot seem to find an OSHMEM test still hung; the tests I'm finding hung are MPI tests.

Here's bt's from the threads in an MPI_Errhandler_fatal_c:

Thread 1 (the app thread)

#0  0x00000034708accdd in nanosleep () from /lib64/libc.so.6
#1  0x00000034708e1e54 in usleep () from /lib64/libc.so.6
#2  0x00002aaaac3eae4b in OPAL_MCA_PMIX2X_PMIx_Abort (flag=6, 
    msg=0x2aaaabfbc837 'N/A\000', procs=0x0, nprocs=0)
    at client/pmix_client.c:559
#3  0x00002aaaac3ab0f0 in pmix2x_abort (flag=6, msg=0x2aaaabfbc837 'N/A\000', 
    procs=0x0) at pmix2x_client.c:174
#4  0x00002aaaabee4db0 in rte_abort (status=6, report=true)
    at ess_pmi_module.c:487
#5  0x00002aaaabecc5e3 in orte_errmgr_base_abort (error_code=6, fmt=0x0)
    at base/errmgr_base_fns.c:224
#6  0x00002aaaabecd2a0 in abort_peers (procs=0x1ff7a00, num_procs=31, 
    error_code=6) at errmgr_default_app.c:199
#7  0x00002aaaab4f259c in try_kill_peers (comm=0x1ff38e0, errcode=6)
    at runtime/ompi_mpi_abort.c:109
#8  0x00002aaaab4f2840 in ompi_mpi_abort (comm=0x1ff38e0, errcode=6)
    at runtime/ompi_mpi_abort.c:193
#9  0x00002aaaab4d9426 in backend_fatal (
    type=0x2aaaab6f3f90 'communicator\000', comm=0x1ff38e0, 
    name=0x1ff3970 'MPI COMMUNICATOR 3 DUP FROM 0\000', 
    error_code=0x7fffffffc9a8, arglist=0x7fffffffc8b8)
    at errhandler/errhandler_predefined.c:392
#10 0x00002aaaab4d8898 in ompi_mpi_errors_are_fatal_comm_handler (
    comm=0x7fffffffc9d0, error_code=0x7fffffffc9a8)
    at errhandler/errhandler_predefined.c:70
#11 0x00002aaaab4d8244 in ompi_errhandler_invoke (
    errhandler=0x2aaaab9c2600 <ompi_mpi_errors_are_fatal>, 
    mpi_object=0x1ff38e0, object_type=1, err_code=6, 
    message=0x2aaaab709e11 <FUNC_NAME> 'MPI_Send\000')
    at errhandler/errhandler_invoke.c:58
#12 0x00002aaaab54ced2 in PMPI_Send (buf=0x7fffffffcee4, count=1, 
    type=0x2aaaab9ae940 <ompi_mpi_integer>, dest=32, tag=0, comm=0x1ff38e0)
    at psend.c:70
#13 0x00002aaaab25ef9a in ompi_send_f (buf=0x7fffffffcee4 '\000', 
    count=0x4161f0, datatype=0x41621c, dest=0x61a900 <mpit_>, tag=0x4161b0, 
    comm=0x7fffffffceec, ierr=0x7fffffffcee4) at psend_f.c:78
#14 0x0000000000401ea3 in MAIN__ () at MPI_Errhandler_fatal_f.F:131

I'm guessing that this is the only bt that is interesting / relevant, but I'll post the rest of them, too, just for completeness.

Thread 2

#0  0x00000034708df343 in poll () from /lib64/libc.so.6
#1  0x00002aaaac3570dd in poll_dispatch (base=0x674500, tv=0x2aaaacffae30)
    at poll.c:165
#2  0x00002aaaac34bdbb in opal_libevent2022_event_base_loop (base=0x674500, 
    flags=1) at event.c:1630
#3  0x00002aaaac279e80 in progress_engine (obj=0x673e58)
    at runtime/opal_progress_threads.c:105
#4  0x0000003470c079d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000034708e8b6d in clone () from /lib64/libc.so.6

Thread 3

#0  0x00000034708e9163 in epoll_wait () from /lib64/libc.so.6
#1  0x00002aaaac3481eb in epoll_dispatch (base=0x69cc60, tv=0x2aaaad1fbe30)
    at epoll.c:407
#2  0x00002aaaac34bdbb in opal_libevent2022_event_base_loop (base=0x69cc60, 
    flags=1) at event.c:1630
#3  0x00002aaaac42c07a in progress_engine (obj=0x69cc18)
    at runtime/pmix_progress_threads.c:151
#4  0x0000003470c079d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000034708e8b6d in clone () from /lib64/libc.so.6

Thread 4

#0  0x00000034708e9163 in epoll_wait () from /lib64/libc.so.6
#1  0x00002aaaac7b780d in usdf_fabric_progression_thread (v=0x6fde00)
    at prov/usnic/src/usdf_progress.c:115
#2  0x0000003470c079d1 in start_thread () from /lib64/libpthread.so.0
#3  0x00000034708e8b6d in clone () from /lib64/libc.so.6

Thread 5

#0  0x00000034708e9163 in epoll_wait () from /lib64/libc.so.6
#1  0x00002aaaac7b780d in usdf_fabric_progression_thread (v=0xb84600)
    at prov/usnic/src/usdf_progress.c:115
#2  0x0000003470c079d1 in start_thread () from /lib64/libpthread.so.0
#3  0x00000034708e8b6d in clone () from /lib64/libc.so.6

Thread 6

#0  0x00000034708e9163 in epoll_wait () from /lib64/libc.so.6
#1  0x00002aaaac7b780d in usdf_fabric_progression_thread (v=0x1004d60)
    at prov/usnic/src/usdf_progress.c:115
#2  0x0000003470c079d1 in start_thread () from /lib64/libpthread.so.0
#3  0x00000034708e8b6d in clone () from /lib64/libc.so.6

[rhc54]

I don't know if/when I'l have time to look at this - suggest someone else try to do so.

[jsquyres]

@artpol84 @jjhursey Can one of your PMIX'ers have a look at this problem? It's causing infinite-spinning problems in Cisco MTT runs. It looks like there were two signatures to this problem -- the first appears to be fixed by 55e4fba. But there still appears to be a second problem (detailed above).

[rhc54]

AH @!$#!#%#@! - I see the problem. Fix coming