msg/async/rdma: fixes crash in fio #16957
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
fio creates multiple CephContext in a single process. Crash(es) happen because rdma stack has a global resources that are still used from one ceph context while have already been destroyed by another context. The commit removes global instances of RDMA dispatcher and infiniband and makes them context (rdma stack) specific. Signed-off-by: Adir Lev <adirl@mellanox.com> Signed-off-by: Alex Mikheev <alexm@mellanox.com>
yuyuyu101
reviewed
Aug 10, 2017
| @@ -776,9 +778,45 @@ int Infiniband::MemoryManager::get_send_buffers(std::vector<Chunk*> &c, size_t b | |||
| return send->get_buffers(c, bytes); | |||
| } | |||
|
|
|||
| Infiniband::Infiniband(CephContext *cct, const std::string &device_name, uint8_t port_num) | |||
| : cct(cct), lock("IB lock"), device_name(device_name), port_num(port_num) | |||
| bool Infiniband::init_prereq = false; | |||
| ldout(cct, 20) << __func__ << " ms_async_rdma_enable_hugepage value is: " << cct->_conf->ms_async_rdma_enable_hugepage << dendl; | ||
| if (cct->_conf->ms_async_rdma_enable_hugepage){ | ||
| rc = setenv("RDMAV_HUGEPAGES_SAFE","1",1); | ||
| ldout(cct, 20) << __func__ << " RDMAV_HUGEPAGES_SAFE is set as: " << getenv("RDMAV_HUGEPAGES_SAFE") << dendl; |
There was a problem hiding this comment.
level 0 is prefer, it's an important info to know
| assert(tx_cq); | ||
| rx_cq = global_infiniband->create_comp_queue(cct, rx_cc); | ||
| rx_cq = get_stack()->get_infiniband().create_comp_queue(cct, rx_cc); | ||
| assert(rx_cq); | ||
|
|
||
| t = std::thread(&RDMADispatcher::polling, this); | ||
| } | ||
|
|
||
| void RDMADispatcher::polling_stop() | ||
| { |
|
I will test locally |
|
@yuyuyu101 thanks. One remaining problem is that rx buffer pool is still a singleton. However there are pending changes that address this in https://github.com/Mellanox/ceph/commits/luminous-rdma |
yuyuyu101
approved these changes
Aug 11, 2017
|
replaced by #16981 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
@yuyuyu101 this pr fixes the same problem as PR #16893
However instead of using ceph context singleton both infiniband and dispatcher are made part of the
RDMA stack. So each ceph context will have a completely separate resources.
It is also better for the performance because there are less qps that are attached to the same SRQ.
fio creates multiple CephContext in a single process.
Crash(es) happen because rdma stack has a global resources that
are still used from one ceph context while have already been destroyed
by another context.
The commit removes global instances of RDMA dispatcher and infiniband
and makes them context (rdma stack) specific.
Signed-off-by: Adir Lev adirl@mellanox.com
Signed-off-by: Alex Mikheev alexm@mellanox.com