mon: don't set last_osd_report when the pg stats msg is ignored #12975
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In some cases, this may lead to mon wrongly marking an osd down because of no pg stats after a specified time period. Signed-off-by: Zhiqiang Wang <zhiqiang@xsky.com>
liewegas
approved these changes
Jan 18, 2017
tchaikov
reviewed
Jan 19, 2017
| if (!stats->get_orig_source().is_osd() || | ||
| !mon->osdmon()->osdmap.is_up(from) || | ||
| stats->get_orig_source_inst() != mon->osdmon()->osdmap.get_inst(from)) { | ||
| dout(1) << " ignoring stats from non-active osd." << dendl; | ||
| return false; | ||
| } | ||
|
|
||
| last_osd_report[from] = ceph_clock_now(); |
There was a problem hiding this comment.
In some cases
@wonzhq what is the case exactly? for example,
- an osd is marked down by monitor and then
- we received a straying pg stat message from it
last_osd_reportis marked with the time stamp.- after a while, a new osd joined in, and monitor assigned it the osd id of the previously marked down osd, // but the leader calls
check_osd_map()after the map is committed, andcheck_osd_map()will clear the last_osd_report for that osd. - in the
tick()the newly joined osd could be wrongly mark down? (but i doubt. see above)
There was a problem hiding this comment.
check_osd_map could return earlier without clearing last_osd_report if osdmap is not readable or pgmap is not writeable. This is a long time ago fixed bug, and the log file has been removed. I can't verify.
There was a problem hiding this comment.
@wonzhq okay, just wanted to understand if we need to backport this fix or not.
tchaikov
approved these changes
Jan 19, 2017
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In some cases, this may lead to mon wrongly marking an osd down
because of no pg stats after a specified time period.
Signed-off-by: Zhiqiang Wang zhiqiang@xsky.com