Is it possible to communicate using RDMA from linux to windows?

[JibinNajeeb]

I tried 2 possible options
Option 1
I tried the below commands
In Linux
./fi_pingpong -p verbs -e msg
In Windows
fi_pingpong.exe -p netdir -e msg 172.16.1.210

This command shows nothing and never exits. I suppose windows can communicate with linux through networkdirect, but it is not happening here

Option 2
I tried the below commands
In Linux
./fi_pingpong -p verbs -e msg
In Windows
fi_pingpong.exe -p verbs -e msg 172.16.1.210

This provided the below result in windows

I thought verbs wont work in windows, but in libfabric it is working, Can I rely verbs as provider for windows

[shefty]

Yes.

The verbs provider was recently updated to run over network direct. That change (I didn't make this) was done specifically to support interoperability between linux and windows systems. The plan is to remove the netdir provider and keep only verbs (over both libibverbs and networkdirect), since verbs is actively maintained.

[shefty]

Btw, the wire protocols used by the verbs provider and netdir providers are not compatible. I'm referring to the protocols added by the providers themselves.

[JibinNajeeb]

Hi Thanks for the reply. While trying the rdm_rma_event app between linux and windows, Im getting the below error

[shefty]

That test isn't supported by the rxm layer, which ends up being layered over verbs here. You can check fabtests/test_configs/../exclude files for tests that aren't supported

[JibinNajeeb]

I tried creating a sample server(linux) and client(windows) with verbs as provider. The client will send text file to server, anything below 128kb is working, but above that it is showing the error reading cq (-259)[Image also attached].

The same case is working if both server and client are running in the same OS(ie. win-win, linux-linux).

What is it I'm missing?

[shefty]

On Linux, 128k is the default cross-over size between different transfer protocols (eager vs. segmentation-and-reassembly vs. rendezvous). I don't know if Windows follows the same defaults. (My guess would have been yes, but given the failure, maybe not.) You could try changing FI_OFI_RXM_EAGER_LIMIT and FI_OFI_RXM_SAR_LIMIT to different values and see if the failure aligns with those values. These values would need to be set on both sides. The one that aligns with 128k is FI_OFI_RXM_SAR_LIMIT.

[JibinNajeeb]

Thanks, when setting these values FI_OFI_RXM_EAGER_LIMIT and FI_OFI_RXM_SAR_LIMIT it is working fine. This is working for fi_send and fi_recv.
But while testing the rma application(fi_write and fi_read we are receiving the error, please check below screen(The connection success in linux is socket connection only)

[JibinNajeeb]

Also, issue is there when testing the fabtests app, rma_bw.exe(windows) and fi_rma_bw.exe(linux), This time we are getting cq_readerr 5. Please check below screen