osd/PG: discard msgs from down peers #17217
Conversation
| // repop can be discarded by that replica OSD, because the primary resets the | ||
| // connection to it when handling the new osdmap marking it down, and also | ||
| // resets the messenger sesssion when the replica reconnects. to avoid the | ||
| // out-of-order replies, the messages from that replica should be discarded. |
There was a problem hiding this comment.
Can you explain where the out-of-order reply comes from? AFAICS it should be fine to pretend we are still living in the past since the PG hasn't advanced yet. If the peer reconnects, it will get rejected... and if the peer restarts, any message it sends will be marked with a later epoch and we won't process it until we advance maps.
There was a problem hiding this comment.
@liewegas could you check the comments in http://tracker.ceph.com/issues/19605 ? i put some analysis out there.
If the peer reconnects, it will get rejected...
no, it does not get rejected.
src/osd/PG.cc
Outdated
| // connection to it when handling the new osdmap marking it down, and also | ||
| // resets the messenger sesssion when the replica reconnects. to avoid the | ||
| // out-of-order replies, the messages from that replica should be discarded. | ||
| if (osd->get_osdmap()->get_down_at(from) >= m->map_epoch) |
There was a problem hiding this comment.
This asserts exists(osd)... which might not be true. We need to check for that, too!
|
Okay, this seems like a decent workaround. The real problem is in the messenger/osd combination, though. The RESETSESSION in the context of osds is just.. flawed. We really want it to permanently poison the peer session so that it can't reconnect at all, but the reset isn't sufficient to do that (and it's mostly unused for osd<->osd communication already). I think the solution involves elevating the session management out of Messenger and into OSD so that the old and new connection can be disambiguated more clearly, but that's a big project. For now, this will do. |
if a repop is replied after a replica goes down in a new osdmap, and before the pg advances to this new osdmap, the repop replies before this repop can be discarded by that replica OSD, because the primary resets the connection to it when handling the new osdmap marking it down, and also resets the messenger sesssion when the replica reconnects. to avoid the out-of-order replies, the messages from that replica should be discarded. Fixes: http://tracker.ceph.com/issues/19605 Signed-off-by: Kefu Chai <kchai@redhat.com>
|
@liewegas fixed and repushed. |
yuriw
added
wip-yuri3-testing
wip-sage-testing
and removed
wip-yuri3-testing
wip-sage-testing
labels
if a repop is replied after a replica goes down in a new osdmap, and
before the pg advances to this new osdmap, the repop replies before this
repop can be discarded by that replica OSD, because the primary resets the
connection to it when handling the new osdmap marking it down, and also
resets the messenger sesssion when the replica reconnects. to avoid the
out-of-order replies, the messages from that replica should be discarded.
Fixes: http://tracker.ceph.com/issues/19605
Signed-off-by: Kefu Chai kchai@redhat.com