Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
msg/async/rdma: make Infiniband can be forkable #13525
Mar 2, 2017
Hi @yuyuyu101 ,
While working on the RDMA-CM, I hit 2 problems due to the pre/post fork:
This all makes me think if it isn't a sign that that the pre/post fork solution is not the right one, and that the first access to RDMA resources should be done after the daemonize().
What do you think about it? Do you have any suggestions?
Some more details about the problems:
I tried to workaround it using a patch to librdmacm, to make rdma_free_devices() free the resources and close the devices. But then I hit the second problem. I need to close the RDMAServerSocket's that were opened before the fork and are using the old ibv_context. and recreate it after the fork, which is a mess.
Yes, I started doing that, and it seems to be working - although it makes things more complicated - what if the port will get occupied between the pre_fork and the post_fork?
With that we could live, but my main concern is still: librdmacm doesn't have the option to close devices. So I can't actually close IB devices on the pre_fork.
The solutions that I thought for that, are all bad:
Currently I'm working with a modified librdmacm, so I be able to continue develop - but I don't have a solution.
Is there another reason for that?
@amirv hmm, it's odd that rdma-cm doesn't support closing device... if so, it's really a problem. We don't want to rely on a developing feature.
The last option is we delay listen after forking.. maybe we could add "start" interface to NetworkStack and let asyncmessenger call start when ready
Yeh, it is ugly - don't know why it was done like that. Anyway, first time a process calls rdma_get_devices() it will call ucma_init() which will open the devices. ucma_uninit() will only be called on the shared library destructor. No way exists to call it directly.
This should be perfect
I hope that I explain it accuratly:
This is the theoretical part (which @orendu and @Adirl know better than me) - in practice, calls to rdma_() and ibv_() functions failed, unless I used the patched librdmacm which actually close the device in pre_fork and open it again in post_fork.
we could implement prefork/postfork now. according to "is_ready" flag, decide to do actual listen or just queue it wait for ready. this is not a perfect way but I think it could work