Skip to content

Commit

Permalink
[#729] improvement: use foreach when iterate over Roaring64Navigabl…
Browse files Browse the repository at this point in the history
…eMap for better performance (#730)

### What changes were proposed in this pull request?

use foreach when iterate over Roaring64NavigableMap for better performance

### Why are the changes needed?

From the doc of RoaringBitmap/RoaringBitmap#44, it will reduce GC footprint and be faster when using the `foreach`

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Don't need.
  • Loading branch information
zuston committed Mar 17, 2023
1 parent d398148 commit d82e10d
Show file tree
Hide file tree
Showing 2 changed files with 4 additions and 11 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,6 @@
import java.util.List;

import com.google.common.collect.Lists;
import org.roaringbitmap.longlong.LongIterator;
import org.roaringbitmap.longlong.Roaring64NavigableMap;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
Expand Down Expand Up @@ -156,10 +155,7 @@ public List<ShuffleDataSegment> split(ShuffleIndexResult shuffleIndexResult) {

private List<Long> getExpectedTaskIds(Roaring64NavigableMap expectTaskIds) {
List<Long> taskIds = new ArrayList<>();
LongIterator iterator = expectTaskIds.getLongIterator();
while (iterator.hasNext()) {
taskIds.add(iterator.next());
}
expectTaskIds.forEach(value -> taskIds.add(value));
return taskIds;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,6 @@
import com.google.common.collect.Range;
import com.google.common.collect.Sets;
import org.apache.commons.collections.CollectionUtils;
import org.roaringbitmap.longlong.LongIterator;
import org.roaringbitmap.longlong.Roaring64NavigableMap;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
Expand Down Expand Up @@ -441,15 +440,13 @@ public byte[] getFinishedBlockIds(String appId, Integer shuffleId, Set<Integer>
// filter the specific partition blockId in the bitmap to the resultBitmap
protected Roaring64NavigableMap getBlockIdsByPartitionId(Set<Integer> requestPartitions,
Roaring64NavigableMap bitmap, Roaring64NavigableMap resultBitmap) {
LongIterator iter = bitmap.getLongIterator();
long mask = (1L << Constants.PARTITION_ID_MAX_LENGTH) - 1;
while (iter.hasNext()) {
long blockId = iter.next();
final long mask = (1L << Constants.PARTITION_ID_MAX_LENGTH) - 1;
bitmap.forEach(blockId -> {
int partitionId = Math.toIntExact((blockId >> Constants.TASK_ATTEMPT_ID_MAX_LENGTH) & mask);
if (requestPartitions.contains(partitionId)) {
resultBitmap.addLong(blockId);
}
}
});
return resultBitmap;
}

Expand Down

0 comments on commit d82e10d

Please sign in to comment.