mon/OSDMonitor: fix process osd failure #12938
Conversation
|
@tchaikov some cleanup, pls review, thanks! |
| if (ls.front()) | ||
| mon->no_reply(ls.front()); | ||
| ls.pop_front(); | ||
| MonOpRequestRef record_op = fi.cancel_report(reporter); |
There was a problem hiding this comment.
if osd failed finally, this may make these reporters cannot receive lastest
update right away,
could you elaborate this a little bit?
There was a problem hiding this comment.
although one reporter has canceled fail, but if some more osds report this osd failed after that, and suppose that mon has enough reporter to mark this osd down, then, in process_failures(), it will call take_all_failures() which get all reporter's report_message, only if reporter has recorded report_message mon will send lastest osdmap to that reporter. so if we drop report_message of all another reporters when one cancel report may cause these reporter cannot receive the update right away, although it will finally acknowledge it by peer's sharing osdmap
There was a problem hiding this comment.
note to myself and posterity:
if osd.42 is reported to as "failed" by its peer. and later the peer receives a ping_reply from osd.42 somehow, so it sends a failure message to monitor to cancel the previous failure report. as a side-effect, on the monitor handling this failure report, all report messages from other OSD peers were also erased from the failure_info. but their osd ids and failed_since are kept around in the failure_info. and they will be taken into consideration in OSDMonitor::check_failure() even the report messages are reset.
if osd.42 is finally marked down, all the osds not reverting their failure report will not be updated with the latest osdmap. but we could.
the related change was introduced by ad12b0d, so we don't leak the failure messages before canceling the report. but failure_report and failure_reporter_t are able to take care of their life cycles just fine without the fix.
| // calculate failure time | ||
| utime_t now = ceph_clock_now(g_ceph_context); | ||
| utime_t failed_since = | ||
| m->get_recv_stamp() - utime_t(m->failed_for, 0); |
LiumxNL
force-pushed
the
fix-process-osd-failure
branch
2 times, most recently
from
2479eb2
to
a0958bc
Compare
liewegas
changed the title
| if (ls.front()) | ||
| mon->no_reply(ls.front()); | ||
| ls.pop_front(); | ||
| MonOpRequestRef record_op = fi.cancel_report(reporter); |
There was a problem hiding this comment.
nit, could you s/record_op/report_op/? then it's good to qa run.
|
modulo the nit, lgtm. just needs to s/record_op/report_op/ before the merge, i think. |
…sonable if osd failed finally, this may make these reporters cannot receive lastest update right away, besides, it's not effective to make a traverse of all reporters Signed-off-by: Mingxin Liu <mingxin@xsky.com>
Signed-off-by: Mingxin Liu <mingxin@xsky.com>
LiumxNL
force-pushed
the
fix-process-osd-failure
branch
from
a0958bc
to
0ec21a5
Compare
|
@tchaikov updated. |
No description provided.