Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

mon: MonClient may hang on pinging an unresponsive monitor #9259

Merged
merged 2 commits into from Aug 5, 2016
Merged

mon: MonClient may hang on pinging an unresponsive monitor #9259

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
eec8491
mon: MonClient may hang on pinging monitor forever
xiexingguo May 23, 2016
9f9a8aa
mon: shutdown finisher in a more graceful way
xiexingguo May 23, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/mon/MonClient.h
View file
Open in desktop
Expand Up @@ -72,7 +72,7 @@ struct MonClientPinger : public Dispatcher {
int ret = 0;
while (!done) {
ret = ping_recvd_cond.WaitUntil(lock, until);
if (ret == -ETIMEDOUT)
if (ret == ETIMEDOUT)
break;
}
return ret;
Expand Down
6 changes: 4 additions & 2 deletions src/test/pybind/test_rados.py
View file
Open in desktop
Expand Up @@ -143,8 +143,10 @@ def test_ping_monitor(self):
cmd = {'prefix': 'mon dump', 'format':'json'}
ret, buf, out = self.rados.mon_command(json.dumps(cmd), b'')
for mon in json.loads(buf.decode('utf8'))['mons']:
buf = json.loads(self.rados.ping_monitor(mon['name']))
assert buf.get('health')
while True:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you elaborate a little bit on this change?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See below：
I retested this above test case that failed these changes and it passed(http://daisycloud.org:9091/xxg-2016-06-13_15:24:56-rados-wip-xxg-testing---basic-plana/468/).
So the problem I guess is it happened that one of the monitors was unable to up and respond in 300s, which wouldn't be a problem before this change as we would wait forever(so perhaps this test case was going to die under this case?)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tchaikov To be more specific:
before this change this test case won't be a problem as we'll ping and wait for response forever.
But this change allow monitor to return error on ETIMEDOUT which will fail the test...

buf = json.loads(self.rados.ping_monitor(mon['name']))
if buf.get('health'):
break

def test_create(self):
self.rados.create_pool('foo')
Expand Down