msg/simple: wait dispatch_queue until all pipes closed

[yuyuyu101]

Now we use dispatch_queue.wait to wait for SimpleMessenger shutdown,
but we need to ensure DispatchQueue can process event after Accepter down,
Otherwise accepter may continue to accept new connection which may queue
new item. so we can't rely on DispatchQueue now.

Introduce stop_cond and stop flag to indicate this function like
AsyncMessenger did

Signed-off-by: Haomai Wang haomai@xsky.com

[gregsfortytwo]

If we don't stop the dispatch_queue until after all the pipes are closed...can't the DIspatcher try to re-open Connections in order to deliver message responses?
If we're worried about accepting new connections, maybe we need to re-order the Accepter shutdown, but that doesn't mean stopping the DispatchQueue way at the end of the process.

[yuyuyu101]

@gregsfortytwo good comment, updated!

[gregsfortytwo]

Okay, this is probably good now.

[tchaikov]

2016-07-01T23:28:37.185 INFO:tasks.ceph.mon.b.mira089.stderr:/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.0-252-gae36581/src/msg/simple/SimpleMessenger.cc: 230: FAILED assert(!cleared)
2016-07-01T23:28:37.190 INFO:tasks.ceph.mon.b.mira089.stderr: ceph version v11.0.0-252-gae36581 (ae3658114102a34fd78b7ba68640558df3c111fd)
2016-07-01T23:28:37.193 INFO:tasks.ceph.mon.b.mira089.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x95) [0x1378e27]
2016-07-01T23:28:37.197 INFO:tasks.ceph.mon.b.mira089.stderr: 2: (SimpleMessenger::reaper()+0x35d) [0x1470187]
2016-07-01T23:28:37.201 INFO:tasks.ceph.mon.b.mira089.stderr: 3: (SimpleMessenger::wait()+0xacb) [0x1473671]
2016-07-01T23:28:37.203 INFO:tasks.ceph.mon.b.mira089.stderr: 4: (main()+0x4872) [0x109a4df]
2016-07-01T23:28:37.205 INFO:tasks.ceph.mon.b.mira089.stderr: 5: (__libc_start_main()+0xf5) [0x7f51e6c18ec5]
2016-07-01T23:28:37.207 INFO:tasks.ceph.mon.b.mira089.stderr: 6: ceph-mon() [0x10937f9]

see

• http://qa-proxy.ceph.com/teuthology/kchai-2016-07-01_23:17:08-rados-wip-kefu-testing---basic-mira/287140/teuthology.log
• http://qa-proxy.ceph.com/teuthology/kchai-2016-07-01_23:17:08-rados-wip-kefu-testing---basic-mira/287128/teuthology.log

dropped from the test branch for now

[yuriw]

@yuyuyu101 @tchaikov I took it for testing as see lots of

11.0.0/src/msg/simple/SimpleMessenger.cc: 230: FAILED assert(!cleared)
2016-07-14T18:12:58.818 INFO:tasks.ceph.mon.b.smithi018.stderr: ceph version v11.0.0-633-g061bf04 (061bf04)
2016-07-14T18:12:58.823 INFO:tasks.ceph.mon.b.smithi018.stderr: 1: (ceph::ceph_assert_fail(char const, char const, int, char const*)+0x85) [0x8517e5]
2016-07-14T18:12:58.827 INFO:tasks.ceph.mon.b.smithi018.stderr: 2: (SimpleMessenger::reaper()+0x6ff) [0x8d033f]
2016-07-14T18:12:58.829 INFO:tasks.ceph.mon.b.smithi018.stderr: 3: (SimpleMessenger::wait()+0x49f) [0x8d0b7f]
2016-07-14T18:12:58.845 INFO:tasks.ceph.mon.b.smithi018.stderr: 4: (main()+0x268b) [0x5e756b]
2016-07-14T18:12:58.859 INFO:tasks.ceph.mon.b.smithi018.stderr: 5: (__libc_start_main()+0xf5) [0x7fda39754b15]
2016-07-14T18:12:58.867 INFO:tasks.ceph.mon.b.smithi018.stderr: 6: ceph-mon() [0x648a85]
2016-07-14T18:12:58.874 INFO:tasks.ceph.mon.b.smithi018.stderr: NOTE: a copy of the executable, or objdump -rdS <executable> is needed to interpret this.

in run http://pulpito.ceph.com/yuriw-2016-07-14_17:00:02-rados-wip-yuri-testing2_2016_7_14-distro-basic-smithi/
removed testing and needs-qa tags

[yuyuyu101]

retest this please

[yuriw]

http://pulpito.ceph.com/yuriw-2016-08-26_22:32:43-rados-wip-yuri-testing_2016_08_26-distro-basic-smithi