Add memory limiter for the add entry request to avoid OOM #3139
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The test come with next commit |
zymap
changed the title
|
@zymap Have you tried configuring backpressure for the BK client and server? I think it covers your case. |
|
@dlg99 Thanks for your information. But looks like the maxAddsInProgressLimit only applied on the v3 request? |
|
I tried the backpressure with V2 protocol. It doesn't work. Because it is a server limitation, when there has a server slow to respond, the other server will respond successfully, but the client will hold the entry buffer until the slowest response is returned. Same issue: https://lists.apache.org/thread/r2yh6gxgnhzzz35o63db040wt2t6o108 |
eolivelli
requested changes
Jun 12, 2022
| @@ -103,6 +106,7 @@ public class BookieClientImpl implements BookieClient, PerChannelBookieClientFac | |||
| private final BookieAddressResolver bookieAddressResolver; | |||
|
|
|||
| private final long bookieErrorThresholdPerInterval; | |||
| private Optional<MemoryLimitController> memoryLimitController; | |||
| @@ -378,6 +414,31 @@ public void safeRun() { | |||
| } | |||
| } | |||
|
|
|||
| private static class WriteAndFlushCallbackImpl implements WriteAndFlushCallback { | |||
There was a problem hiding this comment.
This should be probably recycled
| private Optional<WriteAndFlushCallback> setMemoryLimit(final long entryId, final long entrySize) throws InterruptedException { | ||
| if (getMemoryLimitController().isPresent()) { | ||
| MemoryLimitController mlc = getMemoryLimitController().get(); | ||
| mlc.reserveMemory(entrySize); |
There was a problem hiding this comment.
We should handle this with a timeout otherwise we may block forever?
| @@ -200,6 +200,10 @@ public class ClientConfiguration extends AbstractConfiguration<ClientConfigurati | |||
| protected static final String CLIENT_CONNECT_BOOKIE_UNAVAILABLE_LOG_THROTTLING = | |||
| "clientConnectBookieUnavailableLogThrottling"; | |||
|
|
|||
| // client memory limit options | |||
| protected static final String CLIENT_MEMORY_LIMIT_ENABLED = "clientMemoryLimitEnabled"; | |||
There was a problem hiding this comment.
It looks like this is only for writes.
We should reflect this in the name
bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BookieClientImpl.java
Outdated
Show resolved
Hide resolved
|
@hangc0276 I don't think that the issue is resolved. @zymap what do you think ? |
--- *Motivation* When there has a bookie is slowly to respond the add entry request, and the AQ is smaller than WQ, the client will hole the entry buffer and that will cause the memory won't be released. More context: apache/pulsar#14861 *Modifications* Add the memory limit for the client to send add request.
zymap
force-pushed
the
client-memory-limit
branch
from
fa285bf
to
a55b91b
Compare
|
The test output looks like works as expected, I will try to add more tests and clean the code. logs |
zymap
changed the title
hangc0276
reviewed
Oct 18, 2022
| } | ||
|
|
||
| public long getWriteMemoryLowWaterMark() { | ||
| return getInt(WRITE_MEMORY_LOW_WATER_MARK, 64 * 1024 * 1024); |
| } | ||
|
|
||
| public long getWriteMemoryHighWaterMark() { | ||
| return getInt(WRITE_MEMORY_HIGH_WATER_MARK, 256 * 1024 * 1024); |
| */ | ||
| public class WriteWaterMark { | ||
| private static final long DEFAULT_LOW_WATER_MARK = 1610612736L; // 1.5GB | ||
| private static final long DEFAULT_HIGH_WATER_MARK = 2147483648L; // 2GB |
There was a problem hiding this comment.
Why the default lowWaterMark=64MB and HighWaterMark=256MB in ClientConfiguration, but default lowWaterMark=1.5GB and HighWaterMark=2GB in WriteWaterMark?
hangc0276
reviewed
Oct 18, 2022
| @Slf4j | ||
| public class WriteMemoryCounter { | ||
| private final WriteWaterMark writeWaterMark; | ||
| private AtomicLong sizeCounter = new AtomicLong(0); |
|
|
||
| public void decrementPendingWriteBytes(long size) { | ||
| long usage = sizeCounter.addAndGet(-size); | ||
| log.info("decrement the size to {}", usage); |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Master issue: #3231
Motivation
When there has a bookie is slow to respond the add entry request,
and the AQ is smaller than WQ, the client will hole the entry buffer
and that will cause the memory won't be released.
More context:
apache/pulsar#14861
Modifications
Add the memory limit for the client to the send add request.