msg/async/rdma: Introduce RDMAConnMgr + Debug prints #14201

Adirl · 2017-03-28T17:13:13Z

No description provided.

Adirl · 2017-03-28T17:14:22Z

@yuyuyu101 @amirv @orendu @saritz @DanielBar-On

patchset number 4
Introduce RDMAConnTCP + Debug prints

yuyuyu101 · 2017-03-28T17:33:27Z

src/msg/async/rdma/RDMAConnectedSocketImpl.cc

  notify_fd = dispatcher->register_qp(qp, this);
  dispatcher->perf_logger->inc(l_msgr_rdma_created_queue_pair);
  dispatcher->perf_logger->inc(l_msgr_rdma_active_queue_pair);
 }

+RDMAConnTCP::~RDMAConnTCP()


this should be RDMAConnTCP.cc

yuyuyu101 · 2017-03-28T17:33:46Z

src/msg/async/rdma/RDMAConnectedSocketImpl.cc

@@ -88,7 +103,7 @@ void RDMAConnectedSocketImpl::get_wc(std::vector<ibv_wc> &w)
  w.swap(wc);
 }

-int RDMAConnectedSocketImpl::activate()
+int RDMAConnTCP::activate()


yuyuyu101 · 2017-03-28T17:33:56Z

src/msg/async/rdma/RDMAConnectedSocketImpl.cc

    submit(false);
  }
  active = true;

  return 0;
 }

-int RDMAConnectedSocketImpl::try_connect(const entity_addr_t& peer_addr, const SocketOptions &opts) {
+int RDMAConnTCP::try_connect(const entity_addr_t& peer_addr, const SocketOptions &opts) {


yuyuyu101 · 2017-03-28T17:34:01Z

src/msg/async/rdma/RDMAConnectedSocketImpl.cc

-void RDMAConnectedSocketImpl::handle_connection() {
-  ldout(cct, 20) << __func__ << " QP: " << my_msg.qpn << " tcp_fd: " << tcp_fd << " fd: " << notify_fd << dendl;
-  int r = infiniband->recv_msg(cct, tcp_fd, peer_msg);
+void RDMAConnTCP::handle_connection() {


yuyuyu101 · 2017-03-28T17:34:12Z

src/msg/async/rdma/RDMAConnectedSocketImpl.cc

@@ -575,7 +590,7 @@ void RDMAConnectedSocketImpl::fin() {
  }
 }

-void RDMAConnectedSocketImpl::cleanup() {
+void RDMAConnTCP::cleanup() {


yuyuyu101 · 2017-03-28T17:34:43Z

src/msg/async/rdma/RDMAConnectedSocketImpl.h

 class RDMAConnectedSocketImpl : public ConnectedSocketImpl {
+protected:


one space before "protected"

yuyuyu101 · 2017-03-28T17:35:04Z

src/msg/async/rdma/RDMAConnectedSocketImpl.h

 };

 class RDMAServerSocketImpl : public ServerSocketImpl {
+protected:


yuyuyu101 · 2017-03-28T17:35:21Z

src/msg/async/rdma/RDMAServerSocketImpl.cc

@@ -34,7 +40,7 @@ RDMAServerSocketImpl::RDMAServerSocketImpl(CephContext *cct, Infiniband* i, RDMA
  ibdev->init(ibport);
 }

-int RDMAServerSocketImpl::listen(entity_addr_t &sa, const SocketOptions &opt)
+int RDMAServerConnTCP::listen(entity_addr_t &sa, const SocketOptions &opt)


move to tcp .cc

yuyuyu101 · 2017-03-28T17:37:21Z

src/msg/async/rdma/RDMAConnectedSocketImpl.h

+  virtual void cleanup() = 0;
+  virtual int try_connect(const entity_addr_t&, const SocketOptions &opt) = 0;
+  virtual ssize_t submit(bool more);
+  virtual int remote_qpn() = 0;


is this worth to define a virtual interface? I think we could have a variable in RDMAConnectedSocketImpl

Correct - will change

yuyuyu101 · 2017-03-28T17:40:23Z

src/msg/async/rdma/RDMAConnTCP.h

+class RDMAWorker;
+class RDMADispatcher;
+
+class RDMAConnTCP : public RDMAConnectedSocketImpl {


Maybe we define a RDMAConnMgr to encapsulate management info exchanges

Inherit from RDMAConnectedSocketImpl makes it unclear. Because tcp or rdmacm is only used for management, right?

ack - adding RDMAConnMgr

amirv · 2017-03-29T06:26:25Z

this should be RDMAConnTCP.cc

I did it on purpose - to make the review easier (and rebase if needed).
If you don't mind, I prefer leaving this patch as is, and once it is merged I will send a cleanup commit to move code into the right files.

Encapsulate all connection establishment stuff in a new class - RDMAConnMgr and make it a friend class of RDMAConnectedSocketImpl. This class will be inherited for every type of connection establishment - Currently only TCP is supported, very soon CM will be added too. RDMAServerConnImpl which only handle connection establishment became an abstract class and RDMAServerConnTCP is inherting it for connections of type TCP. Some of the code was left in its original file and place, and therefore it looks misplaced. This was done to make it easier to review and rebase. Once it is accepted a cleanup patch will be sent to move the code into the right place. Issue: 995322 Change-Id: I8b0e163525ec80c2452f4b6481bf696968cc1e51 Signed-off-by: Amir Vadai <amir@vadai.me>

Setting RDMA_DEBUG to 1, will enable debug prints for every ibv_* function called. This makes it easier to debug RDMA issues. Issue: 995322 Change-Id: I49d327d5d4386b44264f5619d50f2dbc7d76dcae Signed-off-by: Amir Vadai <amir@vadai.me>

Adirl · 2017-03-29T14:21:00Z

pushed new patchset and changed PR name
tested on vstart + traffic

yuyuyu101

Nice refactor!

tchaikov · 2017-03-30T04:54:25Z

@Adirl and @yuyuyu101 this breaks master, see https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos7,DIST=centos7,MACHINE_SIZE=huge/6048//consoleFull

[ 38%] Building CXX object src/CMakeFiles/common-objs.dir/msg/async/net_handler.cc.o
In file included from /mnt/jenkins/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.0.0-1939-g4100803/rpm/el7/BUILD/ceph-12.0.0-1939-g4100803/src/msg/async/rdma/RDMAStack.h:30:0,
                 from /mnt/jenkins/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.0.0-1939-g4100803/rpm/el7/BUILD/ceph-12.0.0-1939-g4100803/src/msg/async/Stack.cc:22:
/mnt/jenkins/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.0.0-1939-g4100803/rpm/el7/BUILD/ceph-12.0.0-1939-g4100803/src/msg/async/rdma/Infiniband.h:21:27: fatal error: rdma/rdma_cma.h: No such file or directory
 #include <rdma/rdma_cma.h>
                           ^
compilation terminated.

tchaikov · 2017-03-30T04:55:39Z

and the jenkins "make check" of this PR failed also, see https://jenkins.ceph.com/job/ceph-pull-requests/21036/

tchaikov · 2017-03-30T05:00:44Z

#14245 is posted to revert the offending commit.

yuyuyu101 · 2017-03-30T05:23:08Z

@Adirl plz proposal again

Adirl · 2017-03-30T06:24:27Z

Checking and push new PR
Sorry

amirv · 2017-03-30T07:25:41Z

@yuyuyu101 ,
Sorry, an unused 'include "rdma_cma.h"' slipped into the patch.

We'll send a new PR soon...

@Adirl @orendu

yuyuyu101 reviewed Mar 28, 2017

View reviewed changes

liewegas added common rdma labels Mar 29, 2017

amirv added 2 commits March 29, 2017 17:18

msg/async/rdma: Debug prints for ibv_*

7d1475f

Setting RDMA_DEBUG to 1, will enable debug prints for every ibv_* function called. This makes it easier to debug RDMA issues. Issue: 995322 Change-Id: I49d327d5d4386b44264f5619d50f2dbc7d76dcae Signed-off-by: Amir Vadai <amir@vadai.me>

Adirl force-pushed the rdma-cm-4 branch from d30ad1b to 7d1475f Compare March 29, 2017 14:19

Adirl changed the title ~~msg/async/rdma: Introduce RDMAConnTCP + Debug prints~~ msg/async/rdma: Introduce RDMAConnMgr + Debug prints Mar 29, 2017

yuyuyu101 approved these changes Mar 30, 2017

View reviewed changes

yuyuyu101 merged commit 4100803 into ceph:master Mar 30, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

msg/async/rdma: Introduce RDMAConnMgr + Debug prints #14201

msg/async/rdma: Introduce RDMAConnMgr + Debug prints #14201

Adirl commented Mar 28, 2017

Adirl commented Mar 28, 2017

yuyuyu101 Mar 28, 2017

yuyuyu101 Mar 28, 2017

yuyuyu101 Mar 28, 2017

yuyuyu101 Mar 28, 2017

yuyuyu101 Mar 28, 2017

yuyuyu101 Mar 28, 2017

yuyuyu101 Mar 28, 2017

amirv Mar 29, 2017

yuyuyu101 Mar 28, 2017

yuyuyu101 Mar 28, 2017

amirv Mar 29, 2017

yuyuyu101 Mar 28, 2017

yuyuyu101 Mar 28, 2017

amirv Mar 29, 2017

amirv commented Mar 29, 2017 •

edited

Adirl commented Mar 29, 2017

yuyuyu101 left a comment

tchaikov commented Mar 30, 2017

tchaikov commented Mar 30, 2017

tchaikov commented Mar 30, 2017

yuyuyu101 commented Mar 30, 2017

Adirl commented Mar 30, 2017

amirv commented Mar 30, 2017

		class RDMAConnectedSocketImpl : public ConnectedSocketImpl {
		protected:

msg/async/rdma: Introduce RDMAConnMgr + Debug prints #14201

msg/async/rdma: Introduce RDMAConnMgr + Debug prints #14201

Conversation

Adirl commented Mar 28, 2017

Adirl commented Mar 28, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amirv commented Mar 29, 2017 • edited

Adirl commented Mar 29, 2017

yuyuyu101 left a comment

Choose a reason for hiding this comment

tchaikov commented Mar 30, 2017

tchaikov commented Mar 30, 2017

tchaikov commented Mar 30, 2017

yuyuyu101 commented Mar 30, 2017

Adirl commented Mar 30, 2017

amirv commented Mar 30, 2017

amirv commented Mar 29, 2017 •

edited