I think I found the problem.

I noticed the bml_endpoint was retrieved with mca_bml_get_endpoint() right after cuda_need_buffers(). That function seems to call add_proc() if the endpoint is not there yet.

In btl_array_find(), the only reason why it would crash seems to be the bml_endpoint not being valid. And indeed, we get the bml_endpoint without handling this add_proc() case. So I replaced the code by bml_get_endpoint() and to my greatest surprise, the bug disappeared (at least for the test I was working on).

Here is the patch :

diff --git a/ompi/mca/pml/ob1/pml_ob1_cuda.c b/ompi/mca/pml/ob1/pml_ob1_cuda.c
index 12ad396..8f3985a 100644
--- a/ompi/mca/pml/ob1/pml_ob1_cuda.c
+++ b/ompi/mca/pml/ob1/pml_ob1_cuda.c
@@ -156,8 +156,7 @@ int mca_pml_ob1_cuda_need_buffers(void * rreq,
                                   mca_btl_base_module_t* btl)
 {
     mca_pml_ob1_recv_request_t* recvreq = (mca_pml_ob1_recv_request_t*)rreq;
-    mca_bml_base_endpoint_t* bml_endpoint =
-        (mca_bml_base_endpoint_t*)recvreq->req_recv.req_base.req_proc->proc_endpoints[OMPI_PROC_ENDPOINT_TAG_BML];
+    mca_bml_base_endpoint_t* bml_endpoint = mca_bml_base_get_endpoint (recvreq->req_recv.req_base.req_proc);
     mca_bml_base_btl_t *bml_btl = mca_bml_base_btl_array_find(&bml_endpoint->btl_send, btl);

     /* A btl could be in the rdma list but not in the send list so check there also */

Unless you object within two hours, I'll push this to master so that we have MTT results tomorrow.

Fix CUDA support in v2.0.0 #1402

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions