Skip to content

HDDS-16089. Improve MPU table evaluation efficiency in KeyLifeCycleService - #10949

Open
priyeshkaratha wants to merge 2 commits into
apache:masterfrom
priyeshkaratha:HDDS-16089
Open

HDDS-16089. Improve MPU table evaluation efficiency in KeyLifeCycleService#10949
priyeshkaratha wants to merge 2 commits into
apache:masterfrom
priyeshkaratha:HDDS-16089

Conversation

@priyeshkaratha

@priyeshkaratha priyeshkaratha commented Aug 4, 2026

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

KeyLifecycleService.processMultipartUploads iterated every MPU in a bucket and immediately called OMMultipartUploadUtils.getMultipartOpenKey plus openKeyTable.get(...) before evaluating age or prefix. On buckets
with many active uploads this wasted one key-format resolution and one RocksDB read per MPU, most of which would have been eliminated by the age/prefix check.

Additionally, "orphan" MPUs entries present in multipartInfoTable but whose openKeyTable entry was already removed (e.g. by the old OpenKeyCleanupService before HDDS-9017) were logged as a WARN and skipped entirely, causing them to accumulate in the table forever even when a lifecycle rule plainly covered them.

What is the link to the Apache JIRA

HDDS-16089

How was this patch tested?

Two new integration tests created by claude is added to TestKeyLifecycleService.Normal

Test What it verifies
testOrphanMpuAbortedByAgeAndPrefixRule Orphan MPU is aborted alongside a normal MPU when a prefix-only rule matches
testOrphanMpuNotAbortedByTagOnlyRule Orphan MPU is left untouched when the only applicable rule requires a tag filter

@priyeshkaratha priyeshkaratha changed the title HDDS-16089. Improve MPU table evaluation efficiency in LifeCycleService HDDS-16089. Improve MPU table evaluation efficiency in KeyLifeCycleService Aug 4, 2026
@priyeshkaratha
priyeshkaratha marked this pull request as ready for review August 5, 2026 06:07
}

if (openKeyInfo == null) {
for (OmLCRule rule : ruleList) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This re-scan looks unreachable. We only reach here with matchingRule == null, which means the first loop above already found no tag-free rule passing age+prefix (otherwise matchedRuleWithoutTags is set and this whole read block is skipped). This loop tests the same !rule.isTagEnable() && passesAgeAndPrefix(...) condition, so matchingRule stays null and we always fall through to the skip at the if (matchingRule == null) below — the scheduling abort log a few lines down is never hit.

The orphan tag-free case is actually handled by that first loop (it sets matchedRuleWithoutTags and skips the open-key read entirely, which is why testOrphanMpuAbortedByAgeAndPrefixRule passes). Could we drop this loop (and the unreachable success log) and just keep the skip + continue for the orphan-with-only-tag-rules case?

@rich7420 rich7420 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@priyeshkaratha thanks for the patch!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants