RDMA is slow for weight sync

[casteryh]

🐛 Describe the bug

RDMA is slow

Ongoing:
meta-pytorch/torchstore#53

Versions

No response

[casteryh]

with this PR meta-pytorch/torchstore#55, we can reduce update weights time from 100s to <20s.
However, push_weights performance remains similar with concurrent puts.