HDDS-16089. Improve MPU table evaluation efficiency in KeyLifeCycleService - #10949
HDDS-16089. Improve MPU table evaluation efficiency in KeyLifeCycleService#10949priyeshkaratha wants to merge 2 commits into
Conversation
| } | ||
|
|
||
| if (openKeyInfo == null) { | ||
| for (OmLCRule rule : ruleList) { |
There was a problem hiding this comment.
This re-scan looks unreachable. We only reach here with matchingRule == null, which means the first loop above already found no tag-free rule passing age+prefix (otherwise matchedRuleWithoutTags is set and this whole read block is skipped). This loop tests the same !rule.isTagEnable() && passesAgeAndPrefix(...) condition, so matchingRule stays null and we always fall through to the skip at the if (matchingRule == null) below — the scheduling abort log a few lines down is never hit.
The orphan tag-free case is actually handled by that first loop (it sets matchedRuleWithoutTags and skips the open-key read entirely, which is why testOrphanMpuAbortedByAgeAndPrefixRule passes). Could we drop this loop (and the unreachable success log) and just keep the skip + continue for the orphan-with-only-tag-rules case?
rich7420
left a comment
There was a problem hiding this comment.
@priyeshkaratha thanks for the patch!
What changes were proposed in this pull request?
KeyLifecycleService.processMultipartUploadsiterated every MPU in a bucket and immediately calledOMMultipartUploadUtils.getMultipartOpenKeyplusopenKeyTable.get(...)before evaluating age or prefix. On bucketswith many active uploads this wasted one key-format resolution and one RocksDB read per MPU, most of which would have been eliminated by the age/prefix check.
Additionally, "orphan" MPUs entries present in
multipartInfoTablebut whoseopenKeyTableentry was already removed (e.g. by the oldOpenKeyCleanupServicebefore HDDS-9017) were logged as a WARN and skipped entirely, causing them to accumulate in the table forever even when a lifecycle rule plainly covered them.What is the link to the Apache JIRA
HDDS-16089
How was this patch tested?
Two new integration tests created by claude is added to
TestKeyLifecycleService.NormaltestOrphanMpuAbortedByAgeAndPrefixRuletestOrphanMpuNotAbortedByTagOnlyRule