msg/async: set nonce before starting the workers #12390

tchaikov · 2016-12-08T12:46:39Z

otherwise workers will respond with different nonces to peers.
and remove nonce from Processor. as there is only one nonce for each
Messenger at a given time.

Signed-off-by: Kefu Chai kchai@redhat.com

ghost · 2016-12-08T13:54:57Z

jenkins test this please (jenkins long paths hit again http://tracker.ceph.com/issues/15249)

yuyuyu101 · 2016-12-08T14:54:27Z

src/msg/async/AsyncMessenger.cc

-      p->start();
-    } else {
+    ldout(cct, 10) << __func__ << " will try " << bind_addr
+		   << " and avoid ports " << new_avoid << dendl;


move this out of loop?

wjwithagen · 2016-12-08T16:43:13Z

@tchaikov
I will testrun tonight.

tchaikov · 2016-12-08T16:53:58Z

@wjwithagen please hold on until the tests pass. i am still fixing this PR, it fails 2 tests.

tchaikov · 2016-12-08T18:54:36Z

@wjwithagen fixed. good to test.

tchaikov · 2016-12-08T18:56:03Z

changelog

should be if (r) in AsyncMessenger::bind() and AsyncMessenger::rebind(), not if (!r).

wjwithagen · 2016-12-08T18:58:22Z

@tchaikov
One of them is socket path too long??
I have a feeling for already quite some time that they are better of in /tmp/var/run or some place like that or /tmp/td/....

otherwise workers will respond with difference nonces to peers. and remove nonce from Processor. as there is only one nonce for each Messenger at a given time. Signed-off-by: Kefu Chai <kchai@redhat.com>

tchaikov · 2016-12-08T19:06:09Z

changelog

start() all bound processors instead of the processors[0..i). they are equivalent, but the new version is easier to understand.

tchaikov · 2016-12-08T19:08:37Z

@wjwithagen no, i didn't take that one into account. the two failed tests were the osd-osd-markdown.sh and cephtool-test-mon.sh. but i believe that they should pass now.

wjwithagen · 2016-12-08T20:03:34Z

@tchaikov
Oke will grab the code and test

wjwithagen · 2016-12-08T21:00:01Z

Still the same problem:
cephtool-test-mon.sh freezes on
ceph-master/build/bin/ceph tell osd.0 version
repeating over and over in the log:

2016-12-08 21:44:04.450362 b9fad00  1 -- 127.0.0.1:6800/96528 --> 127.0.0.1:7204/0 -- pg_stats(0 pgs tid 97 v 0) v1 -- 0xc29b180 con 0x0
2016-12-08 21:44:04.655291 bb9df80  1 -- 127.0.0.1:6800/96528 <== mon.2 127.0.0.1:7204/0 367 ==== pg_stats_ack(0 pgs tid 97) v1 ==== 4+0+0 (0 0 0) 0xc61e400 con 0xbbce000
2016-12-08 21:44:06.325820 b9f8d80  1 -- 127.0.0.1:6800/1096528 >> - conn(0xc65e800 :6800 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=16 -
2016-12-08 21:44:06.326674 b9f8d80  1 -- 127.0.0.1:6800/1096528 >> - conn(0xc65e800 :6800 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 cs=0 l=0).read_bulk peer close file descriptor 16
2016-12-08 21:44:06.326773 b9f8d80  1 -- 127.0.0.1:6800/1096528 >> - conn(0xc65e800 :6800 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 cs=0 l=0).read_until read failed
2016-12-08 21:44:06.326803 b9f8d80  1 -- 127.0.0.1:6800/1096528 >> - conn(0xc65e800 :6800 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 cs=0 l=0)._process_connection read peer banner and addr failed
2016-12-08 21:44:06.326920 b9f8d80  0 -- 127.0.0.1:6800/1096528 >> - conn(0xc65e800 :6800 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 cs=0 l=0).fault with nothing to send and in the half  accept state just closed
2016-12-08 21:44:06.327034 b9f8d80  1 -- 127.0.0.1:6800/1096528 reap_dead start

So the pid is 96528, as seen from talking to mon.2.
But the other connection has its nonce set 1000000 higher from a rebind.

However if I run ceph from again the similar setup from the commandline is produces the following error:

2016-12-08 21:51:24.829097 812c6fe00  0 -- 127.0.0.1:0/2757266831 >> 127.0.0.1:6800/96528 conn(0x815426000 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY=0 cs=0 l=1)._process_connection connect claims to be 127.0.0.1:6800/1096528 not 127.0.0.1:6800/96528 - wrong node!
2016-12-08 21:51:25.043623 812c6fe00  0 -- 127.0.0.1:0/2757266831 >> 127.0.0.1:6800/96528 conn(0x815426000 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY=0 cs=0 l=1)._process_connection connect claims to be 127.0.0.1:6800/1096528 not 127.0.0.1:6800/96528 - wrong node!
2016-12-08 21:51:25.470788 812c6fe00  0 -- 127.0.0.1:0/2757266831 >> 127.0.0.1:6800/96528 conn(0x815426000 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY=0 cs=0 l=1)._process_connection connect claims to be 127.0.0.1:6800/1096528 not 127.0.0.1:6800/96528 - wrong node!
2016-12-08 21:51:26.272885 812c6fe00  0 -- 127.0.0.1:0/2757266831 >> 127.0.0.1:6800/96528 conn(0x815426000 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY=0 cs=0 l=1)._process_connection connect claims to be 127.0.0.1:6800/1096528 not 127.0.0.1:6800/96528 - wrong node!
2016-12-08 21:51:27.876517 812c6fe00  0 -- 127.0.0.1:0/2757266831 >> 127.0.0.1:6800/96528 conn(0x815426000 :-1 s=STATE_CONNECTING_WAIT_BANNER_AND_IDENTIFY=0 cs=0 l=1)._process_connection connect claims to be 127.0.0.1:6800/1096528 not 127.0.0.1:6800/96528 - wrong node!

The pid of the ceph-osd is actually 1 higher than the nonce.???
and these are its sockets:

jenkins  ceph-osd   96529 14 tcp4   *:6800                *:*
jenkins  ceph-osd   96529 15 tcp4   127.0.0.1:6800        *:*
jenkins  ceph-osd   96529 21 stream /home/jenkins/workspace/ceph-master/build/src/test/td/t-7202/out/osd.0.asok
jenkins  ceph-osd   96529 32 tcp4   127.0.0.1:62506       127.0.0.1:7204
jenkins  ceph-osd   96529 41 tcp4   127.0.0.1:52840       127.0.0.1:6806
jenkins  ceph-osd   96529 43 tcp4   127.0.0.1:62581       127.0.0.1:6808
jenkins  ceph-osd   96529 45 tcp4   127.0.0.1:43692       127.0.0.1:6807
jenkins  ceph-osd   96529 46 tcp4   127.0.0.1:6804        *:*
jenkins  ceph-osd   96529 47 tcp4   127.0.0.1:34029       127.0.0.1:6810
jenkins  ceph-osd   96529 48 tcp4   127.0.0.1:6805        *:*
jenkins  ceph-osd   96529 50 tcp4   127.0.0.1:59044       127.0.0.1:6811
jenkins  ceph-osd   96529 57 tcp4   127.0.0.1:6804        127.0.0.1:33938
jenkins  ceph-osd   96529 58 tcp4   127.0.0.1:6805        127.0.0.1:24705
jenkins  ceph-osd   96529 59 tcp4   127.0.0.1:6800        127.0.0.1:13204
jenkins  ceph-osd   96529 62 tcp4   127.0.0.1:6805        127.0.0.1:40651
jenkins  ceph-osd   96529 64 tcp4   127.0.0.1:6804        127.0.0.1:25652
jenkins  ceph-osd   96529 65 tcp4   127.0.0.1:6800        127.0.0.1:35355

Haven't looked into the new code, but it suffers from exactly the same 'feature' ....

tchaikov · 2016-12-09T02:27:21Z

@wjwithagen yeah, your issue is not root-caused yet. and again, i think the nonce of external messenger should not change once the osd-ceph is up and running.

tchaikov · 2016-12-12T16:03:09Z

http://pulpito.ceph.com/kchai-2016-12-10_12:27:10-rados-wip-kefu-testing---basic-mira/

tchaikov added the bug-fix label Dec 8, 2016

tchaikov requested a review from yuyuyu101 December 8, 2016 12:46

yuyuyu101 reviewed Dec 8, 2016

View reviewed changes

yuyuyu101 approved these changes Dec 8, 2016

View reviewed changes

tchaikov force-pushed the wip-start-after-setting-nonce branch from 3aeefb2 to ff37bca Compare December 8, 2016 15:22

tchaikov force-pushed the wip-start-after-setting-nonce branch from ff37bca to ac9e2fc Compare December 8, 2016 18:54

tchaikov added the needs-qa label Dec 8, 2016

msg/async: set nonce before starting the workers

aac1a3e

otherwise workers will respond with difference nonces to peers. and remove nonce from Processor. as there is only one nonce for each Messenger at a given time. Signed-off-by: Kefu Chai <kchai@redhat.com>

tchaikov force-pushed the wip-start-after-setting-nonce branch from ac9e2fc to aac1a3e Compare December 8, 2016 19:03

tchaikov self-assigned this Dec 8, 2016

tchaikov added the wip-kefu-testing label Dec 9, 2016

tchaikov merged commit 9418e07 into ceph:master Dec 12, 2016

tchaikov deleted the wip-start-after-setting-nonce branch December 12, 2016 16:03

tchaikov assigned yuyuyu101 and unassigned tchaikov Dec 12, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

msg/async: set nonce before starting the workers #12390

msg/async: set nonce before starting the workers #12390

tchaikov commented Dec 8, 2016 •

edited

ghost commented Dec 8, 2016

yuyuyu101 Dec 8, 2016

tchaikov Dec 8, 2016

wjwithagen commented Dec 8, 2016

tchaikov commented Dec 8, 2016

tchaikov commented Dec 8, 2016

tchaikov commented Dec 8, 2016

wjwithagen commented Dec 8, 2016

tchaikov commented Dec 8, 2016

tchaikov commented Dec 8, 2016 •

edited

wjwithagen commented Dec 8, 2016

wjwithagen commented Dec 8, 2016

tchaikov commented Dec 9, 2016 •

edited

tchaikov commented Dec 12, 2016

msg/async: set nonce before starting the workers #12390

msg/async: set nonce before starting the workers #12390

Conversation

tchaikov commented Dec 8, 2016 • edited

ghost commented Dec 8, 2016

yuyuyu101 Dec 8, 2016

Choose a reason for hiding this comment

tchaikov Dec 8, 2016

Choose a reason for hiding this comment

wjwithagen commented Dec 8, 2016

tchaikov commented Dec 8, 2016

tchaikov commented Dec 8, 2016

tchaikov commented Dec 8, 2016

wjwithagen commented Dec 8, 2016

tchaikov commented Dec 8, 2016

tchaikov commented Dec 8, 2016 • edited

wjwithagen commented Dec 8, 2016

wjwithagen commented Dec 8, 2016

tchaikov commented Dec 9, 2016 • edited

tchaikov commented Dec 12, 2016

tchaikov commented Dec 8, 2016 •

edited

tchaikov commented Dec 8, 2016 •

edited

tchaikov commented Dec 9, 2016 •

edited