CUDA-aware one sided put does not work on master

[rolfv]

With the new osc/rdma component, the CUDA-aware support does not work anymore on onesided put operations. Interestingly, it seems to work on get. I need to investigate but I want to record this bug so that I do not forget it. Also, it would be nice for this to work prior to the osc/rdma coming over to v2.x which I assume it will. Is that true @hjelmn ?

[hjelmn]

Yes. This is intended for 2.x. Let me know if you can determine why puts are failing. It would be great if the new component can make use of the GPU direct support.

[rolfv]

I think they are a few things happening here and maybe we can work our way through them. First, I will try to get osc/pt2pt to continue to work on master. Does that seem OK @hjelm or is it going to go away? It may be nice to keep it for a little while so in case we do not get CUDA-aware working with osc/rdma we will have a fallback. If so, then any idea why I would run into this issue? I only see this with CUDA-aware. Although interesting I do not see this particular failure on 2.x.

[rvandevaart@drossetti-ivy4 nv]$ mpirun --mca osc pt2pt -host drossetti-ivy4,drossetti-ivy5 -np 2 --mca btl self,smcuda,openib osu_get_bw_cuda
# OSU MPI2 MPI_Get Bandwidth Test (Version 2.2)
# Size        Bandwidth (MillionBytes/s) 
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 0 on node drossetti-ivy4 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
[rvandevaart@drossetti-ivy4 nv]$ 

(gdb) where
#0  0x00007f04887951d0 in ?? ()
#1  <signal handler called>
#2  0x00007f0487f87fa1 in ompi_osc_pt2pt_sync_array_peer (rank=1, peers=0xacfef70, nranks=1, peer=0x0)
    at ../../../../../ompi/mca/osc/pt2pt/osc_pt2pt_sync.c:58
#3  0x00007f0487f880cf in ompi_osc_pt2pt_sync_pscw_peer (module=0xad04140, target=1, peer=0x0)
    at ../../../../../ompi/mca/osc/pt2pt/osc_pt2pt_sync.c:89
#4  0x00007f0487f855d2 in osc_pt2pt_incoming_post (module=0xad04140, source=1)
    at ../../../../../ompi/mca/osc/pt2pt/osc_pt2pt_active_target.c:593
#5  0x00007f0487f822b3 in ompi_osc_pt2pt_callback (request=0xa634600)
    at ../../../../../ompi/mca/osc/pt2pt/osc_pt2pt_data_move.c:1636
#6  0x00007f048922341f in ompi_request_complete (request=0xa634600, with_signal=true)
    at ../../../../../ompi/request/request.h:404
#7  0x00007f0489223eb5 in recv_request_pml_complete (recvreq=0xa634600)
    at ../../../../../ompi/mca/pml/ob1/pml_ob1_recvreq.h:199
#8  0x00007f0489224b89 in mca_pml_ob1_recv_frag_callback_match (btl=0xa5f19a0, tag=65 'A', des=0xacd9900, cbdata=0x0)
    at ../../../../../ompi/mca/pml/ob1/pml_ob1_recvfrag.c:245
#9  0x00007f048a5115a4 in btl_openib_handle_incoming (openib_btl=0xa5f19a0, ep=0xa76b140, frag=0xacd9900, byte_len=20)
    at ../../../../../opal/mca/btl/openib/btl_openib_component.c:3090
#10 0x00007f048a512638 in handle_wc (device=0xa5e8980, cq=0, wc=0x7fff02297bb0)
    at ../../../../../opal/mca/btl/openib/btl_openib_component.c:3518
#11 0x00007f048a512c4d in poll_device (device=0xa5e8980, count=2)
    at ../../../../../opal/mca/btl/openib/btl_openib_component.c:3650
#12 0x00007f048a51309a in progress_one_device (device=0xa5e8980)
    at ../../../../../opal/mca/btl/openib/btl_openib_component.c:3760
#13 0x00007f048a513136 in btl_openib_component_progress () at ../../../../../opal/mca/btl/openib/btl_openib_component.c:3783
#14 0x00007f048ee481d3 in opal_progress () at ../../opal/runtime/opal_progress.c:189
#15 0x00007f0487f83ad2 in opal_condition_wait (c=0xad042c8, m=0xad04260) at ../../../../../opal/threads/condition.h:76
#16 0x00007f0487f84d10 in ompi_osc_pt2pt_complete (win=0xad0ad00)
    at ../../../../../ompi/mca/osc/pt2pt/osc_pt2pt_active_target.c:396
#17 0x00007f04901321b0 in PMPI_Win_complete (win=0xad0ad00) at pwin_complete.c:54
#18 0x00000000004018c3 in main (argc=1, argv=0x7fff0229afa8) at osu_get_bw_cuda.c:132
(gdb) up
#1  <signal handler called>
(gdb) up
#2  0x00007f0487f87fa1 in ompi_osc_pt2pt_sync_array_peer (rank=1, peers=0xacfef70, nranks=1, peer=0x0)
    at ../../../../../ompi/mca/osc/pt2pt/osc_pt2pt_sync.c:58
58        if (0 == nranks || (1 == nranks && peers[0]->rank != rank)) {
(gdb) print peers
$1 = (ompi_osc_pt2pt_peer_t **) 0xacfef70
(gdb) print peers[0]
$2 = (ompi_osc_pt2pt_peer_t *) 0x0
(gdb) 

[hjelmn]

osc/pt2pt is a permanent component. It is intended to support one-sided in the absence of RDMA.

[hjelmn]

Also, the failure looks like it is in the new synchronization object which is not yet in 2.x. I will try to take a look tomorrow if you don't get a chance to track this down.

[rolfv]

This change fixed the osc pt2pt problem so that is good. Thanks. There is a way to check if it is a GPU buffer. Where do you think this check would go. I can try and test that out. The function is in opal/mca/common/cuda/common_cuda.c and is called. Actually, by the time we reach the onesided component, we may have already checked and put a flag in the convertor that indicates if it is GPU memory.

[hjelmn]

The check would need to go in ompi_osc_rdma_component_query in osc_rdma_component.c . If the flavor is MPI_WIN_FLAVOR_CREATE check if *base is a cuda pointer.