New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

msg/async/rdma: Add fork safe on RDMA #13740

Merged
merged 1 commit into from Mar 25, 2017

Conversation

Projects
None yet
4 participants
@Adirl

Adirl commented Mar 2, 2017

Using forks with RDMA is dangerous, as memory is registered directly to
hardware. In the special case where the child processes do not need to
do any RDMA, calling ibv_fork_init() will instruct the OS to keep the
physical pages mapped to the parent process, hence allowing it to keep
doing RDMA safely. Should be called before any other RDMA related calls.

Signed-off-by: Sarit Zubakov saritz@mellanox.com

msg/async/rdma: Add fork safe on RDMA
Using forks with RDMA is dangerous, as memory is registered directly to
hardware. In the special case where the child processes do not need to
do any RDMA, calling ibv_fork_init() will instruct the OS to keep the
physical pages mapped to the parent process, hence allowing it to keep
doing RDMA safely. Should be called before any other RDMA related calls.

issue: 972567

Change-Id: I0816f62aadb02d8aa060db0c648198026bd71514
Signed-off-by: Sarit Zubakov <saritz@mellanox.com>
@Adirl

This comment has been minimized.

Adirl commented Mar 2, 2017

@yuyuyu101
pls take a look

@yuyuyu101

This comment has been minimized.

Member

yuyuyu101 commented Mar 2, 2017

since we merged #13525 , I think registered memory is safe now that we dereg memory before fork and reg new memory after fork

@liewegas liewegas added the common label Mar 2, 2017

@Adirl

This comment has been minimized.

Adirl commented Mar 14, 2017

@yuyuyu101
agree no effect expect, still we believe it's is the recommended way when doing verbs

@yuyuyu101

This comment has been minimized.

Member

yuyuyu101 commented Mar 14, 2017

because I heard that calling this will invoke extra overhead for ibverbs calls?

@Adirl

This comment has been minimized.

Adirl commented Mar 14, 2017

from: http://www.rdmamojo.com/2012/05/24/ibv_fork_init/

Calling ibv_fork_init() will reduce performance due to an extra system call for every memory registration, and the extra memory allocated to track memory regions. The precise performance impact depends on the workload and usually will not be significant.

we are not registering memory on the data path (fast path), so i think we are ok,
agree?

@yuyuyu101 yuyuyu101 merged commit 49d0003 into ceph:master Mar 25, 2017

3 checks passed

Signed-off-by all commits in this PR are signed
Details
Unmodifed Submodules submodules for project are unmodified
Details
default Build finished.
Details

@Adirl Adirl deleted the Adirl:forksafe branch Apr 18, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment