osd/scrub: tag replica scrub messages to identify stale events

[ronen-fr]

A lot can happen while a replica is waiting for the backend
to collect the map data. The Primary might, for example, abort
the scrub and start a new one (following no-scrub commands).

The 'token' introduced here tags 'replica scrub resched'
messages with an index value that is modified on each 'release
scrub resources' request from the Primary.

Fixes: https://tracker.ceph.com/issues/52012
Signed-off-by: Ronen Friedman rfriedma@redhat.com

[athanatos]

I don't quite understand the token lifecycle. It doesn't look like it gets encoded in any primary->replica messages, so it's purely used for the replica to cancel any outstanding local events when its reservation is canceled?

[ronen-fr]

I don't quite understand the token lifecycle. It doesn't look like it gets encoded in any primary->replica messages, so it's purely used for the replica to cancel any outstanding local events when its reservation is canceled?

Yes. (And - even internal to the replica, I started by adding it to some of the events, but found out
later it's not needed)

[ronen-fr]

I don't quite understand the token lifecycle. It doesn't look like it gets encoded in any primary->replica messages, so it's purely used for the replica to cancel any outstanding local events when its reservation is canceled?

yes
(we have discussed this issue once, some months ago. At that time - I was under the false assumption that if the Primary is stopping and restarting the scrub, the replica will notice a new interval. So - there are two separate issues to handle:

• how can a replica know that the scrub request is new?, and
• if handling a new request - what enables us to discard the stale internal events.)

[neha-ojha]

I don't quite understand the token lifecycle. It doesn't look like it gets encoded in any primary->replica messages, so it's purely used for the replica to cancel any outstanding local events when its reservation is canceled?

yes
(we have discussed this issue once, some months ago. At that time - I was under the false assumption that if the Primary is stopping and restarting the scrub, the replica will notice a new interval. So - there are two separate issues to handle:

• how can a replica know that the scrub request is new?, and
• if handling a new request - what enables us to discard the stale internal events.)

@ronen-fr how was pre-refactored code handling this? It seems like https://tracker.ceph.com/issues/52012#note-1 was possible even earlier?

[github-actions]

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

[github-actions]

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

[ronen-fr]

I don't quite understand the token lifecycle. It doesn't look like it gets encoded in any primary->replica messages, so it's purely used for the replica to cancel any outstanding local events when its reservation is canceled?

yes
(we have discussed this issue once, some months ago. At that time - I was under the false assumption that if the Primary is stopping and restarting the scrub, the replica will notice a new interval. So - there are two separate issues to handle:

• how can a replica know that the scrub request is new?, and
• if handling a new request - what enables us to discard the stale internal events.)

@ronen-fr how was pre-refactored code handling this? It seems like https://tracker.ceph.com/issues/52012#note-1 was possible even earlier?

(before actual analysis of the code:) I think so. Requires a combination of "scrub abort" and max_osd_scrubs>1.

[github-actions]

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

[ronen-fr]

Initial set of 72 tests is clean:
http://pulpito.front.sepia.ceph.com/?branch=wip-ronenf-scrub-token

[ronen-fr]

Runs:
original: yuriw-2021-09-09_21:20:54-rados-wip-yuri-testing-2021-09-09-1227-distro-basic-smithi (9F, 16D)
rerun: yuriw-2021-09-10_14:25:47-rados-wip-yuri-testing-2021-09-09-1227-distro-basic-smithi (4F, 1D)

Failures:

#6382926: “cephadm/test_dashboard_e2e.sh” - dashboard related failure. Seems to be https://tracker.ceph.com/issues/52417
#6382927: mgr/rook: might be https://tracker.ceph.com/issues/52321
#6382940: ceph_test_objectstore crash. Opened https://tracker.ceph.com/issues/52576
#6382942: seems the same as #6382926

The dead job: test issue (reimaging)