osd: heartbeat peers need to be updated when a new OSD added into an existed cluster #12069
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 2 commits
November 18, 2016 20:01
…already existed cluster Signed-off-by: Pan Liu <pan.liu@istuary.com>
Signed-off-by: Pan Liu <pan.liu@istuary.com>
liewegas
changed the title
liewegas
approved these changes
Nov 18, 2016
|
Note that this only happens if the new OSD gets no PGs mapped to it, which is pretty rare. Still, let's fix it! |
|
@liewegas yes, I'm confusing why io will hang if no pg mapping |
|
the problem solved here is fixing incorrect "up" status in mon I think. @liupan1111 is it right? |
|
@yuyuyu101, at this case when a new osd joins, and then be killed, monitor still treat it as up, So client will still take this osd in its computing scope. |
|
We add osdid +1 and -1 to the peer set so that we ensure we always ping all osds and the graph is fully connected. See maybe_update_heartbeat_peers() |
|
@liupan1111 - I see no tracker ticket reference, it'd be useful if you can link it. |
|
@yuriw , done. Thank you for reminding. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The case is:
There is a storage node which has two osds. nearly one min later, a new OSD joins in. Monitor sees it as "up", and no heatbeat from existing two osds to this new one. When we kill this new OSD, no osd reports this to monitor, so monitor still treats it as "up". Then, if the user invokes fio, io may hang there.
Fixes: http://tracker.ceph.com/issues/18004