rgw: fix versioned bucket index logic so that incomplete index transactions can be fixed #55165
Conversation
|
This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved |
| if (r < 0) | ||
| return r; | ||
| if (exists) { | ||
| r = target->prepare_atomic_modification(dpp, op, false, NULL, NULL, NULL, true, false, y); |
There was a problem hiding this comment.
This prepare_atomic_modification is protected by a check of exists. Yet I don't see a similar protection below in the corresponding call to complete_atomic_modification. Is that an issue or am I missing something?
There was a problem hiding this comment.
Double check my analysis here, but it looks to me like the complete_atomic_modification method is cleaning up the tail of an object and it seems that it's possible for the tail to not be cleaned up (if the process is aborted or killed or the request thread is terminated early in some other way). I was thinking that the subsequent attempts to delete the object should again attempt to delete any tail objects, in case some were left behind. It's a good point though, and a scenario that we should probably test to make sure there are no surprises.
If I can carve out some time, I'd be happy to do a little refactoring here and write some better documentation on how the versioned bucket index is implemented. I'll definitely try. |
|
@cfsnyder can you please rebase? |
…ith existing tooling Adds test cases to reproduce a scenario where leftover versioned bucket index entries are not fixable/removable by existing radosgw-admin commands nor by subsequent delete requests. Signed-off-by: Cory Snyder <csnyder@1111systems.com>
…ctions can be fixed When a versioned bucket index transaction does not complete normally, it can leave behind index entries which cannot be cleaned up by subsequent delete requests or by radosgw-admin commands. These leftover index entries can also cause the bucket object count to be incorrect in a manner that is not fixable by existing tooling. This commit fixes the logic to allow those transactions to be completed by subsequent deletes or by radosgw-admin commands. Fixes: https://tracker.ceph.com/issues/64016 Signed-off-by: Cory Snyder <csnyder@1111systems.com>
cfsnyder
force-pushed
the
wip-64016-cfsnyder
branch
from
77a5bcd
to
b22e202
Compare
|
jenkins test windows |
|
failed qa with ceph_test_cls_rgw failures (example teuthology.log): |
this one corresponds to https://github.com/ceph/ceph/blob/6e39aa0/src/test/cls_rgw/test_cls_rgw.cc#L207-L231 where DEL races with ADD. because the DEL completes first and deletes the entry, the ADD fails afterwards |
|
Thanks @cbodley , I'll be looking into it this week. |
|
@cbodley I'm not seeing how this failing test case represents a realistic scenario. How would a delete happen concurrently with an add for the same object instance? It seems to me that an RGW client would always need to get the response from a PUT to be aware of the instance ID before being able to issue an associated delete, so in practice there should always be a happens-before relationship. Am I missing something? |
|
rgw: fix versioned bucket index logic so that incomplete index transactions can be fixed
When a versioned bucket index transaction does not complete normally,
it can leave behind index entries which cannot be cleaned up
by subsequent delete requests or by radosgw-admin commands. These
leftover index entries can also cause the bucket object count
to be incorrect in a manner that is not fixable by existing tooling.
This commit fixes the logic to allow those transactions to be
completed by subsequent deletes or by radosgw-admin commands.
Fixes: https://tracker.ceph.com/issues/64016
Signed-off-by: Cory Snyder csnyder@1111systems.com
Checklist
Show available Jenkins commands
jenkins retest this pleasejenkins test classic perfjenkins test crimson perfjenkins test signedjenkins test make checkjenkins test make check arm64jenkins test submodulesjenkins test dashboardjenkins test dashboard cephadmjenkins test apijenkins test docsjenkins render docsjenkins test ceph-volume alljenkins test ceph-volume toxjenkins test windowsjenkins test rook e2e