Adaptive: Fix concurrency issue in adaptive allocator#16767
Merged
Conversation
Motivation: We have a few code paths where `Chunk.remainingCapacity()` is called without regard for exclusive access to the given chunk. The `BuddyChunk` implementation unfortunately drained the `freeList` in this call, leading to racy modifications of the `buddies` array, that in turn can manifest as unpredictable bugs. Modification: Make the `BuddyChunk.remainingCapacity` non-modifying. Instead of draining the `freeList`, we sum up the capacity held by it and add it to the capacity accounted for in the `Chunk` super class. This operation needs a `weakPeekReduce` method on the `MpscIntQueue`. This method performs a reduction operation on a given range of entries, but without removing entries or incrementing the consumer index. A regression test is added, which successfully captured the issue before it was fixed. Also did a few test cleanups. Result: No more corruptions in the buddies array. This fixes the rare occurrences of assertion errors like the following: ``` java.lang.AssertionError at io.netty.buffer.AdaptivePoolingAllocator$Magazine.allocate(AdaptivePoolingAllocator.java:916) at io.netty.buffer.AdaptivePoolingAllocator$Magazine.tryAllocate(AdaptivePoolingAllocator.java:854) at io.netty.buffer.AdaptivePoolingAllocator$MagazineGroup.allocate(AdaptivePoolingAllocator.java:421) at io.netty.buffer.AdaptivePoolingAllocator.allocate(AdaptivePoolingAllocator.java:271) ```
chrisvest
added
needs-cherry-pick-4.1
chrisvest
force-pushed
the
4.2-adaptive-fix
branch
from
ca1f910 to
62a3fd2
Compare
normanmaurer
requested changes
May 8, 2026
Member
Author
|
@normanmaurer @franz1981 comments addressed |
normanmaurer
approved these changes
May 8, 2026
Contributor
|
Could not create auto-port PR. |
Contributor
|
Auto-port PR for 5.0: #16777 |
github-actions
Bot
removed
the
needs-cherry-pick-5.0
Member
Author
|
4.1 PR: #16778 |
chrisvest
removed
the
needs-cherry-pick-4.1
chrisvest
added a commit
that referenced
this pull request
May 8, 2026
…16777) Auto-port of #16767 to 5.0 Cherry-picked commit: bd866c3 --- Motivation: We have a few code paths where `Chunk.remainingCapacity()` is called without regard for exclusive access to the given chunk. The `BuddyChunk` implementation unfortunately drained the `freeList` in this call, leading to racy modifications of the `buddies` array, that in turn can manifest as unpredictable bugs. Modification: Make the `BuddyChunk.remainingCapacity` non-modifying. Instead of draining the `freeList`, we sum up the capacity held by it and add it to the capacity accounted for in the `Chunk` super class. This operation needs a `weakPeekReduce` method on the `MpscIntQueue`. This method performs a reduction operation on a given range of entries, but without removing entries or incrementing the consumer index. A regression test is added, which successfully captured the issue before it was fixed. Also did a few test cleanups. Result: No more corruptions in the buddies array. This fixes the rare occurrences of assertion errors like the following: ``` java.lang.AssertionError at io.netty.buffer.AdaptivePoolingAllocator$Magazine.allocate(AdaptivePoolingAllocator.java:916) at io.netty.buffer.AdaptivePoolingAllocator$Magazine.tryAllocate(AdaptivePoolingAllocator.java:854) at io.netty.buffer.AdaptivePoolingAllocator$MagazineGroup.allocate(AdaptivePoolingAllocator.java:421) at io.netty.buffer.AdaptivePoolingAllocator.allocate(AdaptivePoolingAllocator.java:271) ``` Co-authored-by: Chris Vest <christianvest_hansen@apple.com>
chrisvest
added a commit
that referenced
this pull request
May 9, 2026
Motivation: We have a few code paths where `Chunk.remainingCapacity()` is called without regard for exclusive access to the given chunk. The `BuddyChunk` implementation unfortunately drained the `freeList` in this call, leading to racy modifications of the `buddies` array, that in turn can manifest as unpredictable bugs. Modification: Make the `BuddyChunk.remainingCapacity` non-modifying. Instead of draining the `freeList`, we sum up the capacity held by it and add it to the capacity accounted for in the `Chunk` super class. This operation needs a `weakPeekReduce` method on the `MpscIntQueue`. This method performs a reduction operation on a given range of entries, but without removing entries or incrementing the consumer index. A regression test is added, which successfully captured the issue before it was fixed. Also did a few test cleanups. Result: No more corruptions in the buddies array. This fixes the rare occurrences of assertion errors like the following: ``` java.lang.AssertionError at io.netty.buffer.AdaptivePoolingAllocator$Magazine.allocate(AdaptivePoolingAllocator.java:916) at io.netty.buffer.AdaptivePoolingAllocator$Magazine.tryAllocate(AdaptivePoolingAllocator.java:854) at io.netty.buffer.AdaptivePoolingAllocator$MagazineGroup.allocate(AdaptivePoolingAllocator.java:421) at io.netty.buffer.AdaptivePoolingAllocator.allocate(AdaptivePoolingAllocator.java:271) ``` (cherry picked from commit bd866c3)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation:
We have a few code paths where
Chunk.remainingCapacity()is called without regard for exclusive access to the given chunk. TheBuddyChunkimplementation unfortunately drained thefreeListin this call, leading to racy modifications of thebuddiesarray, that in turn can manifest as unpredictable bugs.Modification:
Make the
BuddyChunk.remainingCapacitynon-modifying. Instead of draining thefreeList, we sum up the capacity held by it and add it to the capacity accounted for in theChunksuper class.This operation needs a
weakPeekReducemethod on theMpscIntQueue. This method performs a reduction operation on a given range of entries, but without removing entries or incrementing the consumer index.A regression test is added, which successfully captured the issue before it was fixed.
Also did a few test cleanups.
Result:
No more corruptions in the buddies array.
This fixes the rare occurrences of assertion errors like the following: