-
Notifications
You must be signed in to change notification settings - Fork 28.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-46401][CORE] Use !isEmpty()
on RoaringBitmap
instead of getCardinality() > 0
in RemoteBlockPushResolver
#44347
Conversation
!isEmpty()
on RoaringBitmap instead of getCardinality() > 0
!isEmpty()
on RoaringBitmap instead of getCardinality() > 0
in RemoteBlockPushResolver
@@ -809,7 +809,7 @@ public MergeStatuses finalizeShuffleMerge(FinalizeShuffleMerge msg) { | |||
msg.shuffleMergeId, partition.reduceId); | |||
// This can throw IOException which will marks this shuffle partition as not merged. | |||
partition.finalizePartition(); | |||
if (partition.mapTracker.getCardinality() > 0) { | |||
if (!partition.mapTracker.isEmpty()) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There should only be this one place where getCardinality() > 0
is used for empty-checking on RoaringBitmap
!isEmpty()
on RoaringBitmap instead of getCardinality() > 0
in RemoteBlockPushResolver
!isEmpty()
on RoaringBitmap
instead of getCardinality() > 0
in RemoteBlockPushResolver
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice catch @LuciferYang !
Looks good to me.
Why is this still DRAFT ? :-)
+CC @otterc as well |
Thanks @mridulm ~ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for this change. LGTM
What changes were proposed in this pull request?
This pr use
!mapTracker.isEmpty()
instead ofmapTracker.getCardinality() > 0
inRemoteBlockPushResolver
to reduce some computational costs, as thegetCardinality()
method triggers a recount every time it is called.This change is refer to RoaringBitmap/RoaringBitmap#239 | RoaringBitmap/RoaringBitmap#684
Why are the changes needed?
Use a more appropriate API to reduce the computational cost of empty-checking for RoaringBitmap.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Pass GitHub Actions
Was this patch authored or co-authored using generative AI tooling?
No