BTL TCP abort in MPI_FINALIZE with progress thread

[jsquyres]

On Cisco master MTT runs when running with the BTL TCP progress thread, I'm seeing aborts during MPI_FINALIZE when the TCP BTL progress thread is enabled.

https://mtt.open-mpi.org/index.php?do_redir=2371

The failure signatures are similar to the one shown below; it looks like it's trying to destroy something in btl_tcp_del_procs that doesn't have a refcount of 0:

 Warning :: opal_list_remove_item - the item 0x7c22d0 is not on the list 0x7371d8 
ialltoall: class/opal_list.c:69: opal_list_item_destruct: Assertion `0 == item->opal_list_item_refcount' failed.[mpi001:27496] *** Process received signal ***
[mpi001:27496] Signal: Aborted (6)
[mpi001:27496] Signal code:  (-6)[mpi001:27496] [ 0] /lib64/libpthread.so.0[0x31ec20f710]
[mpi001:27496] [ 1] /lib64/libc.so.6(gsignal+0x35)[0x31ebe32925]
[mpi001:27496] [ 2] /lib64/libc.so.6(abort+0x175)[0x31ebe34105]
[mpi001:27496] [ 3] /lib64/libc.so.6[0x31ebe2ba4e]
[mpi001:27496] [ 4] /lib64/libc.so.6(__assert_perror_fail+0x0)[0x31ebe2bb10]
[mpi001:27496] [ 5] /home/mpiteam/scratches/community/2016-10-19cron/VmSh/installs/3bSr/install/lib/libopen-pal.so.0(+0x4a3e9)[0x2aaaab4fb3e9]
[mpi001:27496] [ 6] /home/mpiteam/scratches/community/2016-10-19cron/VmSh/installs/3bSr/install/lib/libopen-pal.so.0(+0xed79f)[0x2aaaab59e79f]
[mpi001:27496] [ 7] /home/mpiteam/scratches/community/2016-10-19cron/VmSh/installs/3bSr/install/lib/libopen-pal.so.0(mca_btl_tcp_del_procs+0x118)[0x2aaaab59f60b]
[mpi001:27496] [ 8] /home/mpiteam/scratches/community/2016-10-19cron/VmSh/installs/3bSr/install/lib/libmpi.so.0(+0x1460a7)[0x2aaaaabf30a7]
[mpi001:27496] [ 9] /home/mpiteam/scratches/community/2016-10-19cron/VmSh/installs/3bSr/install/lib/libmpi.so.0(mca_pml_ob1_del_procs+0x2b)[0x2aaaaad5a8df]
[mpi001:27496] [10] /home/mpiteam/scratches/community/2016-10-19cron/VmSh/installs/3bSr/install/lib/libmpi.so.0(ompi_mpi_finalize+0x724)[0x2aaaaab344a3]
[mpi001:27496] [11] /home/mpiteam/scratches/community/2016-10-19cron/VmSh/installs/3bSr/install/lib/libmpi.so.0(PMPI_Finalize+0x59)[0x2aaaaab63865]
[mpi001:27496] [12] collective/ialltoall[0x4011bf]
[mpi001:27496] [13] /lib64/libc.so.6(__libc_start_main+0xfd)[0x31ebe1ed1d]
[mpi001:27496] [14] collective/ialltoall[0x400e69]
[mpi001:27496] *** End of error message ***
-------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
---------------------------------------------------------------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 0 on node mpi001 exited on signal 6 (Aborted).
--------------------------------------------------------------------------

@bosilca Could you have a look?

[bosilca]

I run few hundreds tests (ibm/collective/ialltoall) and I can't replicate. I also audit the code, and I can hardly see how this might happen except if we are trying to call btl_del_proc multiple time.

[jsquyres]

@thananon Is this the bug you were referring to in email?

[thananon]

No, I was talking in the case without progress thread.

[ggouaillardet]

@bosilca i am able to reproduce the bug when i run the ialltoallv program in a loop (2 nodes, 16 tasks per node)
i added a few printf here and there, and at this stage, the tcp_btl->tcp_endpoints list is accessed simultaneously by two threads, which likely ends up in a corruption.

i suspect add_procs gets invoked by a thread in the middle of the main thread running del_procs, but i was unable to fully evidence this yet.

[ggouaillardet]

here is a trace that evidence a potential issue.

mca_btl_tcp_add_procs is invoked by two threads at the same time, and though invoked with distinct procs, they both end up opal_list_append() the very same tcp_btl->tcp_endpoints list

i will protect list access with a lock and see how things go.
i will also add some trace to detect whether mca_btl_tcp_add_procs might be invoked at the same time and on the same proc

[bosilca]

We were very careful when we designed the handshake, as now there are multiple path inside. I don't think there is a need for more locks. First, the proc creation happens in mca_btl_tcp_proc_create, where the management of procs is protected behind the mca_btl_tcp_component.tcp_lock lock. As we add the procs in a hash table based on their unique gid, there is no opportunity for collision here.

Once we return from this function, everything else is protected behind the proc lock, as all updates will now affect an existing proc. I don't know where you put your asserts (the line numbers do not match my files at all), so it is difficult to assess how you got to this point. I am trying to reproduce on my own environment.

[ggouaillardet]

here is my current diff (with a bunch of fprintf)

line numbers are different than the ones on the stack trace

diff --git a/opal/class/opal_list.c b/opal/class/opal_list.c
index e0a5112..844a919 100644
--- a/opal/class/opal_list.c
+++ b/opal/class/opal_list.c
@@ -22,6 +22,8 @@
 #include "opal/class/opal_list.h"
 #include "opal/constants.h"

+#include <poll.h>
+
 /*
  *  List classes
  */
@@ -66,6 +68,11 @@ static void opal_list_item_construct(opal_list_item_t *item)
 static void opal_list_item_destruct(opal_list_item_t *item)
 {
 #if OPAL_ENABLE_DEBUG
+    if( 0 != item->opal_list_item_refcount ) {
+        volatile int _dbg = 1;
+        fprintf(stderr, "0 != item->opal_list_item_refcount\n");
+        while (_dbg) poll(NULL, 0, 1);
+    }
     assert( 0 == item->opal_list_item_refcount );
     assert( NULL == item->opal_list_item_belong_to );
 #endif  /* OPAL_ENABLE_DEBUG */
diff --git a/opal/mca/btl/tcp/btl_tcp.c b/opal/mca/btl/tcp/btl_tcp.c
index cb56c10..05aa21e 100644
--- a/opal/mca/btl/tcp/btl_tcp.c
+++ b/opal/mca/btl/tcp/btl_tcp.c
@@ -31,12 +31,15 @@
 #include "opal/mca/mpool/base/base.h"
 #include "opal/mca/mpool/mpool.h"
 #include "opal/mca/btl/base/btl_base_error.h"
+// #include "opal/include/opal/sys/atomic.h"

 #include "btl_tcp.h"
 #include "btl_tcp_frag.h"
 #include "btl_tcp_proc.h"
 #include "btl_tcp_endpoint.h"

+#include <poll.h>
+
 mca_btl_tcp_module_t mca_btl_tcp_module = {
     .super = {
         .btl_component = &mca_btl_tcp_component.super,
@@ -53,6 +56,42 @@ mca_btl_tcp_module_t mca_btl_tcp_module = {
     }
 };

+volatile int32_t in_tcp_endpoints = 0;
+
+static volatile opal_vpid_t assert_vpid = -1;
+static volatile bool assert_wait = false;
+
+static void assert_0(void) {
+    if (0 != opal_atomic_add_32(&in_tcp_endpoints, -1)) {
+        poll(NULL, 0, 2);
+        if (assert_wait) {
+            volatile int _dbg = 0;
+            fprintf(stderr, "attach %d\n", getpid());
+            while (_dbg) poll(NULL, 0, 1);
+        } else {
+            fprintf(stderr, "avoided %d\n", getpid());
+        }
+    }
+}
+
+static void assert_1(opal_proc_t * proc) {
+    if (1 != opal_atomic_add_32(&in_tcp_endpoints, 1)) {
+        poll(NULL, 0, 1);
+        if (NULL != proc && assert_vpid == proc->proc_name.vpid) {
+            volatile int _dbg = 1;
+            assert_wait = true;
+            fprintf(stderr, "attach %d\n", getpid());
+            while (_dbg) poll(NULL, 0, 1);
+        } else {
+            fprintf(stderr, "avoided %d\n", getpid());
+        }
+    } else {
+        if (NULL != proc) {
+            assert_vpid = proc->proc_name.vpid;
+        }
+    }
+}
+
 /**
  *
  */
@@ -117,12 +156,17 @@ int mca_btl_tcp_add_procs( struct mca_btl_base_module_t* btl,
             tcp_endpoint->endpoint_btl = tcp_btl;
             rc = mca_btl_tcp_proc_insert(tcp_proc, tcp_endpoint);
             if(rc != OPAL_SUCCESS) {
+                fprintf(stderr, "F F F %d/%d with error %d\n", i, nprocs, rc);
                 OPAL_THREAD_UNLOCK(&tcp_proc->proc_lock);
                 OBJ_RELEASE(tcp_endpoint);
                 continue;
             }

+            assert_1(opal_proc);
+            OPAL_THREAD_LOCK(&tcp_btl->tcp_lock);
             opal_list_append(&tcp_btl->tcp_endpoints, (opal_list_item_t*)tcp_endpoint);
+            OPAL_THREAD_UNLOCK(&tcp_btl->tcp_lock);
+            assert_0();
         }

         OPAL_THREAD_UNLOCK(&tcp_proc->proc_lock);
@@ -152,7 +196,11 @@ int mca_btl_tcp_del_procs(struct mca_btl_base_module_t* btl,
     for( i = 0; i < nprocs; i++ ) {
         mca_btl_tcp_endpoint_t* tcp_endpoint = endpoints[i];
         if(tcp_endpoint->endpoint_proc != mca_btl_tcp_proc_local()) {
+            assert_1(tcp_endpoint->endpoint_proc->proc_opal);
+            OPAL_THREAD_LOCK(&tcp_btl->tcp_lock);
             opal_list_remove_item(&tcp_btl->tcp_endpoints, (opal_list_item_t*)tcp_endpoint);
+            OPAL_THREAD_UNLOCK(&tcp_btl->tcp_lock);
+            assert_0();
             OBJ_RELEASE(tcp_endpoint);
         }
         opal_progress_event_users_decrement();
@@ -479,13 +527,19 @@ int mca_btl_tcp_finalize(struct mca_btl_base_module_t* btl)
 {
     mca_btl_tcp_module_t* tcp_btl = (mca_btl_tcp_module_t*) btl;
     opal_list_item_t* item;
+    assert_1(NULL);
+    OPAL_THREAD_LOCK(&tcp_btl->tcp_lock);
     for( item = opal_list_remove_first(&tcp_btl->tcp_endpoints);
          item != NULL;
          item = opal_list_remove_first(&tcp_btl->tcp_endpoints)) {
         mca_btl_tcp_endpoint_t *endpoint = (mca_btl_tcp_endpoint_t*)item;
+        OPAL_THREAD_UNLOCK(&tcp_btl->tcp_lock);
         OBJ_RELEASE(endpoint);
         opal_progress_event_users_decrement();
+        OPAL_THREAD_LOCK(&tcp_btl->tcp_lock);
     }
+    OPAL_THREAD_UNLOCK(&tcp_btl->tcp_lock);
+    assert_0();
     free(tcp_btl);
     return OPAL_SUCCESS;
 }
diff --git a/opal/mca/btl/tcp/btl_tcp.h b/opal/mca/btl/tcp/btl_tcp.h
index 7e9d726..e3ca2f3 100644
--- a/opal/mca/btl/tcp/btl_tcp.h
+++ b/opal/mca/btl/tcp/btl_tcp.h
@@ -189,6 +189,9 @@ struct mca_btl_tcp_module_t {
     uint32_t           tcp_ifmask;  /**< BTL interface netmask */
     opal_list_t        tcp_endpoints;
     mca_btl_base_module_error_cb_fn_t tcp_error_cb;  /**< Upper layer error callback */
+    opal_mutex_t tcp_lock;
+    /**< lock to protect against concurrent access to tcp_endpoints */
+
 #if MCA_BTL_TCP_STATISTICS
     size_t tcp_bytes_sent;
     size_t tcp_bytes_recv;
diff --git a/opal/mca/btl/tcp/btl_tcp_component.c b/opal/mca/btl/tcp/btl_tcp_component.c
index 37e58c2..ff23dc9 100644
--- a/opal/mca/btl/tcp/btl_tcp_component.c
+++ b/opal/mca/btl/tcp/btl_tcp_component.c
@@ -17,7 +17,7 @@
  *                         reserved.
  * Copyright (c) 2013-2015 NVIDIA Corporation.  All rights reserved.
  * Copyright (c) 2014-2015 Intel, Inc. All rights reserved.
- * Copyright (c) 2014-2015 Research Organization for Information Science
+ * Copyright (c) 2014-2016 Research Organization for Information Science
  *                         and Technology (RIST). All rights reserved.
  * $COPYRIGHT$
  *
@@ -490,6 +490,7 @@ static int mca_btl_tcp_create(int if_kindex, const char* if_name)
             return OPAL_ERR_OUT_OF_RESOURCE;
         memcpy(btl, &mca_btl_tcp_module, sizeof(mca_btl_tcp_module));
         OBJ_CONSTRUCT(&btl->tcp_endpoints, opal_list_t);
+        OBJ_CONSTRUCT(&btl->tcp_lock, opal_mutex_t);
         mca_btl_tcp_component.tcp_btls[mca_btl_tcp_component.tcp_num_btls++] = btl;

         /* initialize the btl */
@@ -803,9 +804,9 @@ static void mca_btl_tcp_component_event_async_handler(int fd, short unused, void
     opal_event_t* event;
     int rc;

-    rc = read(fd, (void*)&event, sizeof(opal_event_t*));
+    rc = opal_fd_read(fd, sizeof(opal_event_t*), (void*)&event);
     assert( fd == mca_btl_tcp_pipe_to_progress[0] );
-    if( 0 == rc ) {
+    if( OPAL_ERR_TIMEOUT == rc ) {
         /* The main thread closed the pipe to trigger the shutdown procedure */
         opal_thread_t* current_thread = (opal_thread_t*)context;
         (*((int*)current_thread->t_arg)) = 0;
diff --git a/opal/mca/btl/tcp/btl_tcp_proc.c b/opal/mca/btl/tcp/btl_tcp_proc.c
index f7decc4..e6bc018 100644
--- a/opal/mca/btl/tcp/btl_tcp_proc.c
+++ b/opal/mca/btl/tcp/btl_tcp_proc.c
@@ -422,6 +422,7 @@ int mca_btl_tcp_proc_insert( mca_btl_tcp_proc_t* btl_proc,
     memset(proc_data, 0, sizeof(mca_btl_tcp_proc_data_t));

     if (NULL == (proc_hostname = opal_get_proc_hostname(btl_proc->proc_opal))) {
+        fprintf(stderr, "opal_get_proc_hostname(%s) returned NULL\n", OPAL_NAME_PRINT(btl_proc->proc_opal->proc_name));
         return OPAL_ERR_UNREACH;
     }

@@ -444,6 +445,7 @@ int mca_btl_tcp_proc_insert( mca_btl_tcp_proc_t* btl_proc,
     if( NULL == mca_btl_tcp_retrieve_local_interfaces(proc_data) )
         return OPAL_ERR_OUT_OF_RESOURCE;
     if( 0 == proc_data->num_local_interfaces ) {
+        fprintf(stderr, "0 local interfaces for proc %s\n", OPAL_NAME_PRINT(btl_proc->proc_opal->proc_name));
         return OPAL_ERR_UNREACH;
     }

diff --git a/opal/util/proc.c b/opal/util/proc.c
index bf76399..5e7dc53 100644
--- a/opal/util/proc.c
+++ b/opal/util/proc.c
@@ -191,6 +191,7 @@ char* opal_get_proc_hostname(const opal_proc_t *proc)
     /* if it is my own hostname we are after, then just hand back
      * the value in opal_process_info */
     if (proc == opal_proc_my_name) {
+        if (NULL == opal_process_info.nodename) fprintf(stderr, "opal_process_info.nodename is NULL\n");
         return opal_process_info.nodename;
     }

@@ -207,6 +208,7 @@ char* opal_get_proc_hostname(const opal_proc_t *proc)
         return "unknown";  // return something so the caller doesn't segfault
     }

+    if (NULL == proc->proc_hostname) fprintf(stderr, "proc->proc_hostname (%s) is NULL\n", OPAL_NAME_PRINT(proc->proc_name));
     /* user is not allowed to release the data */
     return proc->proc_hostname;
 }

in mca_btl_tcp_add_procs, i previously noted

        for (uint32_t j = 0 ; j < (uint32_t)tcp_proc->proc_endpoint_count ; ++j) {
            tcp_endpoint = tcp_proc->proc_endpoints[j];
            if (tcp_endpoint->endpoint_btl == tcp_btl) {
                existing_found = true;
                break;
            }
        }

        if (!existing_found) {
            /* The btl_proc datastructure is shared by all TCP BTL
             * instances that are trying to reach this destination.
             * Cache the peer instance on the btl_proc.
             */
            tcp_endpoint = OBJ_NEW(mca_btl_tcp_endpoint_t);
            if(NULL == tcp_endpoint) {
                OPAL_THREAD_UNLOCK(&tcp_proc->proc_lock);
                return OPAL_ERR_OUT_OF_RESOURCE;
            }

            tcp_endpoint->endpoint_btl = tcp_btl;
            rc = mca_btl_tcp_proc_insert(tcp_proc, tcp_endpoint);
            if(rc != OPAL_SUCCESS) {
                OPAL_THREAD_UNLOCK(&tcp_proc->proc_lock);
                OBJ_RELEASE(tcp_endpoint);
                continue;
            }

            assert_1(opal_proc);
            OPAL_THREAD_LOCK(&tcp_btl->tcp_lock);
            opal_list_append(&tcp_btl->tcp_endpoints, (opal_list_item_t*)tcp_endpoint);
            OPAL_THREAD_UNLOCK(&tcp_btl->tcp_lock);
            assert_0();
        }

in the second error path, OPAL_THREAD_UNLOCK is followed by continue.
i would have expected a return or no OPAL_THREAD_UNLOCK at all.

in my current environment, races accessing tcp_btl->tcp_endpoints are detected and avoided.

that being said, we occasionally run into the error path i mentionned earlier.
sometimes, no error occurs, but some other times, it causes a crash.
we run into the error path because in opal_get_proc_hostname does

    OPAL_MODEX_RECV_VALUE(ret, OPAL_PMIX_HOSTNAME, &proc->proc_name,
                          (char**)&(proc->proc_hostname), OPAL_STRING);

it returns successfully, and/but proc->proc_hostname is NULL
at this stage, i do note know whether this is a normal behavior or the result of yet an other race.

[ggouaillardet]

@bosilca
answering your previous comment

opal_list_append(&tcp_btl->tcp_endpoints, (opal_list_item_t*)tcp_endpoint);

in mca_btl_tcp_add_procs is protected by the tcp_proc->proc_lock

what i saw when debugging is that tcp_btl->tcp_endpoints is the same in both threads, but tcp_proc is different (and hence each thread owns its own lock, but update a very same shared variable)

[ggouaillardet]

i just edited the inline patch above.
opal_fd_read params were note swapped as expected

[bosilca]

@ggouaillardet thanks for your finding, you are right the tcp_endpoints was unprotected from multiple thread accessing simultaneously while trying to add endpoints pertaining to different processors. I have put together a small patch, and run few hundreds tests successfully. I will run a little more tests and then create a PR.

diff --git a/opal/mca/btl/tcp/btl_tcp.c b/opal/mca/btl/tcp/btl_tcp.c
index cb56c10..a3cfd9d 100644
--- a/opal/mca/btl/tcp/btl_tcp.c
+++ b/opal/mca/btl/tcp/btl_tcp.c
@@ -50,7 +50,8 @@ mca_btl_tcp_module_t mca_btl_tcp_module = {
         .btl_put = mca_btl_tcp_put,
         .btl_dump = mca_btl_base_dump,
         .btl_ft_event = mca_btl_tcp_ft_event
-    }
+    },
+    .tcp_endpoints_mutex = OPAL_MUTEX_STATIC_INIT
 };

 /**
@@ -122,7 +123,9 @@ int mca_btl_tcp_add_procs( struct mca_btl_base_module_t* btl,
                 continue;
             }

+            OPAL_THREAD_LOCK(&tcp_btl->tcp_endpoints_mutex);
             opal_list_append(&tcp_btl->tcp_endpoints, (opal_list_item_t*)tcp_endpoint);
+            OPAL_THREAD_UNLOCK(&tcp_btl->tcp_endpoints_mutex);
         }

         OPAL_THREAD_UNLOCK(&tcp_proc->proc_lock);
@@ -143,12 +146,14 @@ int mca_btl_tcp_add_procs( struct mca_btl_base_module_t* btl,
 }

 int mca_btl_tcp_del_procs(struct mca_btl_base_module_t* btl,
-        size_t nprocs,
-        struct opal_proc_t **procs,
-        struct mca_btl_base_endpoint_t ** endpoints)
+                          size_t nprocs,
+                          struct opal_proc_t **procs,
+                          struct mca_btl_base_endpoint_t ** endpoints)
 {
     mca_btl_tcp_module_t* tcp_btl = (mca_btl_tcp_module_t*)btl;
     size_t i;
+
+    OPAL_THREAD_LOCK(&tcp_btl->tcp_endpoints_mutex);
     for( i = 0; i < nprocs; i++ ) {
         mca_btl_tcp_endpoint_t* tcp_endpoint = endpoints[i];
         if(tcp_endpoint->endpoint_proc != mca_btl_tcp_proc_local()) {
@@ -157,6 +162,7 @@ int mca_btl_tcp_del_procs(struct mca_btl_base_module_t* btl,
         }
         opal_progress_event_users_decrement();
     }
+    OPAL_THREAD_UNLOCK(&tcp_btl->tcp_endpoints_mutex);
     return OPAL_SUCCESS;
 }

@@ -479,6 +485,10 @@ int mca_btl_tcp_finalize(struct mca_btl_base_module_t* btl)
 {
     mca_btl_tcp_module_t* tcp_btl = (mca_btl_tcp_module_t*) btl;
     opal_list_item_t* item;
+
+    /* Don't lock the tcp_endpoints_mutex, at this point a single
+     * thread should be active.
+     */
     for( item = opal_list_remove_first(&tcp_btl->tcp_endpoints);
          item != NULL;
          item = opal_list_remove_first(&tcp_btl->tcp_endpoints)) {
@@ -512,11 +522,13 @@ void mca_btl_tcp_dump(struct mca_btl_base_module_t* base_btl,
     } else if( verbose ) {
         opal_list_item_t *item;

+        OPAL_THREAD_LOCK(&btl->tcp_endpoints_mutex);
         for(item =  opal_list_get_first(&btl->tcp_endpoints);
             item != opal_list_get_end(&btl->tcp_endpoints);
             item = opal_list_get_next(item)) {
             MCA_BTL_TCP_ENDPOINT_DUMP(10, (mca_btl_base_endpoint_t*)item, false, "TCP");
         }
+        OPAL_THREAD_UNLOCK(&btl->tcp_endpoints_mutex);
     }
 #endif /* OPAL_ENABLE_DEBUG && WANT_PEER_DUMP */
 }
diff --git a/opal/mca/btl/tcp/btl_tcp.h b/opal/mca/btl/tcp/btl_tcp.h
index 90d5a43..6f7b48d 100644
--- a/opal/mca/btl/tcp/btl_tcp.h
+++ b/opal/mca/btl/tcp/btl_tcp.h
@@ -170,7 +170,10 @@ struct mca_btl_tcp_module_t {
 #endif
     struct sockaddr_storage tcp_ifaddr; /**< BTL interface address */
     uint32_t           tcp_ifmask;  /**< BTL interface netmask */
+
+    opal_mutex_t       tcp_endpoints_mutex;
     opal_list_t        tcp_endpoints;
+
     mca_btl_base_module_error_cb_fn_t tcp_error_cb;  /**< Upper layer error callback */
 #if MCA_BTL_TCP_STATISTICS
     size_t tcp_bytes_sent;
diff --git a/opal/mca/btl/tcp/btl_tcp_component.c b/opal/mca/btl/tcp/btl_tcp_component.c
index 1fb4f78..f3567db 100644
--- a/opal/mca/btl/tcp/btl_tcp_component.c
+++ b/opal/mca/btl/tcp/btl_tcp_component.c
@@ -474,6 +474,7 @@ static int mca_btl_tcp_create(int if_kindex, const char* if_name)
             return OPAL_ERR_OUT_OF_RESOURCE;
         memcpy(btl, &mca_btl_tcp_module, sizeof(mca_btl_tcp_module));
         OBJ_CONSTRUCT(&btl->tcp_endpoints, opal_list_t);
+        OBJ_CONSTRUCT(&btl->tcp_endpoints_mutex, opal_mutex_t);
         mca_btl_tcp_component.tcp_btls[mca_btl_tcp_component.tcp_num_btls++] = btl;

         /* initialize the btl */

[bosilca]

pull request #2398 should fix this issue.

[jsquyres]

Looks like #2398 was merged; let's close this. Will open a new one if MTT tests continue to fail.