branch-4.1: [Enhancement](memory) Add ConcurrentLong2ObjectHashMap and ConcurrentLong2LongHashMap (#61332)#63797
Merged
Conversation
…Long2LongHashMap (apache#61332) Add two thread-safe primitive-key concurrent hash maps built on fastutil, designed as drop-in replacements for `ConcurrentHashMap<Long, V>` and `ConcurrentHashMap<Long, Long>` in memory-sensitive FE paths. - **`ConcurrentLong2ObjectHashMap<V>`** — replaces `ConcurrentHashMap<Long, V>` - **`ConcurrentLong2LongHashMap`** — replaces `ConcurrentHashMap<Long, Long>` `ConcurrentHashMap<Long, V>` costs ~64 bytes per entry due to Long boxing, Node wrapper, and segment overhead. These fastutil-based maps reduce that to ~16 bytes per entry — a **4x memory reduction**. In Doris FE, several critical data structures use `ConcurrentHashMap<Long, V>` at tablet/partition scale (millions of entries), making this a significant memory optimization opportunity. - **Segment-based locking** (default 16 segments) for concurrent throughput, similar to Java 7's ConcurrentHashMap design - Full `Map` interface compatibility for drop-in replacement - Atomic operations: `putIfAbsent`, `computeIfAbsent`, `replace`, `remove(key, value)` - Thread-safe iteration via snapshot-based `entrySet()`/`keySet()`/`values()` | Collection | Per-entry overhead | 1M entries | |------------|-------------------|------------| | `ConcurrentHashMap<Long, V>` | ~64 bytes | ~61 MB | | `ConcurrentLong2ObjectHashMap<V>` | ~16 bytes | ~15 MB | | `ConcurrentHashMap<Long, Long>` | ~80 bytes | ~76 MB | | `ConcurrentLong2LongHashMap` | ~16 bytes | ~15 MB | - [x] `ConcurrentLong2ObjectHashMapTest` — 432 lines covering put/get/remove, putIfAbsent, computeIfAbsent, replace, concurrent writes from multiple threads, iteration consistency, empty map edge cases - [x] `ConcurrentLong2LongHashMapTest` — 455 lines covering CRUD, default value semantics, concurrent operations, atomic operations, iteration, edge cases 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR backports an FE enhancement introducing two segmented-lock concurrent hash maps with primitive long keys (and either object or long values) built on fastutil, aimed at reducing memory overhead compared to ConcurrentHashMap<Long, …> in large-scale metadata paths.
Changes:
- Add
ConcurrentLong2ObjectHashMap<V>andConcurrentLong2LongHashMapimplementations infe-foundation(fastutil-backed, segmentReentrantReadWriteLock). - Add JUnit tests covering CRUD, atomic ops, concurrency behavior, and Gson serialization round-trips.
- Update Maven module dependencies/comments to include
fastutil-coreinfe-foundationand document classpath priority infe-core.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 12 comments.
Show a summary per file
| File | Description |
|---|---|
| fe/pom.xml | Minor cleanup (removes a comment near the managed fastutil-core dependency). |
| fe/fe-foundation/pom.xml | Adds fastutil-core dependency to support new primitive concurrent maps. |
| fe/fe-foundation/src/main/java/org/apache/doris/foundation/util/ConcurrentLong2ObjectHashMap.java | New segmented-lock primitive-key concurrent map for long -> V. |
| fe/fe-foundation/src/main/java/org/apache/doris/foundation/util/ConcurrentLong2LongHashMap.java | New segmented-lock primitive-key concurrent map for long -> long, including addTo. |
| fe/fe-core/src/test/java/org/apache/doris/common/util/ConcurrentLong2ObjectHashMapTest.java | Adds unit/concurrency/serialization tests for ConcurrentLong2ObjectHashMap. |
| fe/fe-core/src/test/java/org/apache/doris/common/util/ConcurrentLong2LongHashMapTest.java | Adds unit/concurrency/serialization tests for ConcurrentLong2LongHashMap. |
| fe/fe-core/pom.xml | Updates comment to explain keeping fastutil-core as a direct dependency for classpath precedence. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+49
to
+51
| * <p>Iteration methods ({@link #long2ObjectEntrySet()}, {@link #keySet()}, {@link #values()}) | ||
| * return snapshot copies and are weakly consistent. | ||
| * |
Comment on lines
+52
to
+55
| * <p><b>Important:</b> All compound operations from both {@link Long2ObjectMap} and {@link Map} | ||
| * interfaces (computeIfAbsent, computeIfPresent, compute, merge, putIfAbsent, replace, remove) | ||
| * are overridden to ensure atomicity within a segment. | ||
| * |
Comment on lines
+369
to
+372
| public V merge(long key, V value, BiFunction<? super V, ? super V, ? extends V> remappingFunction) { | ||
| Segment<V> seg = segmentFor(key); | ||
| seg.lock.writeLock().lock(); | ||
| try { |
Comment on lines
+433
to
+449
| // ---- Iteration (weakly consistent snapshots) ---- | ||
|
|
||
| @Override | ||
| public ObjectSet<Long2LongMap.Entry> long2LongEntrySet() { | ||
| ObjectOpenHashSet<Long2LongMap.Entry> snapshot = new ObjectOpenHashSet<>(); | ||
| for (Segment seg : segments) { | ||
| seg.lock.readLock().lock(); | ||
| try { | ||
| for (Long2LongMap.Entry entry : seg.map.long2LongEntrySet()) { | ||
| snapshot.add(new AbstractLong2LongMap.BasicEntry(entry.getLongKey(), entry.getLongValue())); | ||
| } | ||
| } finally { | ||
| seg.lock.readLock().unlock(); | ||
| } | ||
| } | ||
| return snapshot; | ||
| } |
Comment on lines
+52
to
+55
| * <p><b>Important:</b> All compound operations from both {@link Long2LongMap} and {@link Map} | ||
| * interfaces (computeIfAbsent, computeIfPresent, compute, merge, mergeLong, putIfAbsent, | ||
| * replace, remove) are overridden to ensure atomicity within a segment. | ||
| */ |
Comment on lines
+18
to
+20
| package org.apache.doris.foundation.util; | ||
|
|
||
| import com.google.gson.Gson; |
Comment on lines
+202
to
+246
| public V putIfAbsent(long key, V value) { | ||
| Objects.requireNonNull(value, "Null values are not permitted"); | ||
| Segment<V> seg = segmentFor(key); | ||
| seg.lock.writeLock().lock(); | ||
| try { | ||
| V existing = seg.map.get(key); | ||
| if (existing != null || seg.map.containsKey(key)) { | ||
| return existing; | ||
| } | ||
| seg.map.put(key, value); | ||
| return null; | ||
| } finally { | ||
| seg.lock.writeLock().unlock(); | ||
| } | ||
| } | ||
|
|
||
| public boolean replace(long key, V oldValue, V newValue) { | ||
| Objects.requireNonNull(newValue, "Null values are not permitted"); | ||
| Segment<V> seg = segmentFor(key); | ||
| seg.lock.writeLock().lock(); | ||
| try { | ||
| V curValue = seg.map.get(key); | ||
| if (!Objects.equals(curValue, oldValue) || (curValue == null && !seg.map.containsKey(key))) { | ||
| return false; | ||
| } | ||
| seg.map.put(key, newValue); | ||
| return true; | ||
| } finally { | ||
| seg.lock.writeLock().unlock(); | ||
| } | ||
| } | ||
|
|
||
| public V replace(long key, V value) { | ||
| Objects.requireNonNull(value, "Null values are not permitted"); | ||
| Segment<V> seg = segmentFor(key); | ||
| seg.lock.writeLock().lock(); | ||
| try { | ||
| if (seg.map.containsKey(key)) { | ||
| return seg.map.put(key, value); | ||
| } | ||
| return null; | ||
| } finally { | ||
| seg.lock.writeLock().unlock(); | ||
| } | ||
| } |
Comment on lines
+204
to
+243
| public long putIfAbsent(long key, long value) { | ||
| Segment seg = segmentFor(key); | ||
| seg.lock.writeLock().lock(); | ||
| try { | ||
| if (seg.map.containsKey(key)) { | ||
| return seg.map.get(key); | ||
| } | ||
| seg.map.put(key, value); | ||
| return defaultReturnValue(); | ||
| } finally { | ||
| seg.lock.writeLock().unlock(); | ||
| } | ||
| } | ||
|
|
||
| public boolean replace(long key, long oldValue, long newValue) { | ||
| Segment seg = segmentFor(key); | ||
| seg.lock.writeLock().lock(); | ||
| try { | ||
| if (seg.map.containsKey(key) && seg.map.get(key) == oldValue) { | ||
| seg.map.put(key, newValue); | ||
| return true; | ||
| } | ||
| return false; | ||
| } finally { | ||
| seg.lock.writeLock().unlock(); | ||
| } | ||
| } | ||
|
|
||
| public long replace(long key, long value) { | ||
| Segment seg = segmentFor(key); | ||
| seg.lock.writeLock().lock(); | ||
| try { | ||
| if (seg.map.containsKey(key)) { | ||
| return seg.map.put(key, value); | ||
| } | ||
| return defaultReturnValue(); | ||
| } finally { | ||
| seg.lock.writeLock().unlock(); | ||
| } | ||
| } |
Comment on lines
+327
to
+331
| @Override | ||
| public V computeIfAbsent(Long key, Function<? super Long, ? extends V> mappingFunction) { | ||
| return computeIfAbsent(key.longValue(), (long k) -> mappingFunction.apply(k)); | ||
| } | ||
|
|
Comment on lines
+334
to
+351
| @Override | ||
| public Long computeIfAbsent(Long key, Function<? super Long, ? extends Long> mappingFunction) { | ||
| long k = key.longValue(); | ||
| Segment seg = segmentFor(k); | ||
| seg.lock.writeLock().lock(); | ||
| try { | ||
| if (seg.map.containsKey(k)) { | ||
| return seg.map.get(k); | ||
| } | ||
| Long newValue = mappingFunction.apply(key); | ||
| if (newValue != null) { | ||
| seg.map.put(k, newValue.longValue()); | ||
| } | ||
| return newValue; | ||
| } finally { | ||
| seg.lock.writeLock().unlock(); | ||
| } | ||
| } |
Contributor
Author
|
run buildall |
Contributor
FE Regression Coverage ReportIncrement line coverage |
yiguolei
approved these changes
May 29, 2026
github-actions
Bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
Contributor
|
PR approved by anyone and no changes requested. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
pick #61332