osd: fix recovery reservation bugs, and implement remote reservation preemption #18485
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Sage Weil <sage@redhat.com>
This now mirrors the backfill approach (e.g., RequestBackfillPrio). Signed-off-by: Sage Weil <sage@redhat.com>
This way me match the terminology used by MRecoveryReserve. It is also a bit more suggestive of primary->replica action, whereas "cancel" could mean replica canceling its grant. Document the meaning in the headers to clarify meaning. Signed-off-by: Sage Weil <sage@redhat.com>
We were sending REJECT if the replica filled up, and the primary would set the BACKFILL_TOOFULL state as a result. Make it an explicit verb for clarity. Signed-off-by: Sage Weil <sage@redhat.com>
If we have granted a remote backfill reservation, and a higher priority request comes in, send a REVOKE message back to the primary and drop the reservation (allowing the higher-priority reservation to be GRANTed). We can only do this if the primary is running new code because it must understand the REVOKE message. Signed-off-by: Sage Weil <sage@redhat.com>
…ition This is easier to follow than canceling the reservation in the next state. Signed-off-by: Sage Weil <sage@redhat.com>
We were keeping our existing recovery reservation slot (with a high priority) and going straight to waiting for backfill reservations on the peers. This is a problem because the reserver thinks we're doing high priority work when we're actually doing lower-priority backfill. Fix by closing out our recovery reservation and going to the WaitLocalBackfillReserved state, where we'll re-request backfill at the appropriate priority. Signed-off-by: Sage Weil <sage@redhat.com>
liewegas
force-pushed
the
wip-remote-res-preemption
branch
2 times, most recently
from
f475ebb
to
3e91fda
Compare
|
Only some of this can be backported because of the protocol changes. See #18498 |
| GRANT = 1, | ||
| RELEASE = 2, | ||
| REQUEST = 0, // primary->replica: please reserve slot | ||
| GRANT = 1, // replica->primary: ok, i reserved it |
There was a problem hiding this comment.
what does "tabs left" mean?
Signed-off-by: Sage Weil <sage@redhat.com>
…backfill We may have log recovery *and* backfill to do, but cease to be degraded as soon as the log recovery portion is done. If that's the case, clear the DEGRADED bit so that the PG state is not misleading. Signed-off-by: Sage Weil <sage@redhat.com>
liewegas
force-pushed
the
wip-remote-res-preemption
branch
from
3e91fda
to
2207607
Compare
|
only non-noisy failure in there was the bluefs bug fixed by #18503 |
dzafman
reviewed
Oct 25, 2017
| @@ -11231,6 +11231,9 @@ bool PrimaryLogPG::start_recovery_ops( | |||
| if (state_test(PG_STATE_RECOVERING)) { | |||
| state_clear(PG_STATE_RECOVERING); | |||
| state_clear(PG_STATE_FORCED_RECOVERY); | |||
| if (get_osdmap()->get_pg_size(info.pgid.pgid) <= acting.size()) { | |||
| state_clear(PG_STATE_DEGRADED); | |||
There was a problem hiding this comment.
FYI, In the future I'm hoping to base PG_STATE_DEGRADED on the num_objects_degraded count.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Bugs:
...and implement remote recovery preemption. With this change we should always be
working on the highest priority recovery, no matter what.
The remaining gap that I see is that there is sometimes work that we could be doing
but aren't because of the ordered primary-then-replicas lock ordering approach.