More memory limiting for HttpPostEmitter #5300
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jon-wei
force-pushed
the
emitter_memory
branch
from
6d59e8a
to
4181785
Compare
himanshug
approved these changes
Jan 26, 2018
gianm
reviewed
Jan 26, 2018
docs/content/configuration/index.md
Outdated
| @@ -222,7 +222,7 @@ The Druid servers [emit various metrics](../operations/metrics.html) and alerts | |||
| |`druid.emitter.http.basicAuthentication`|Login and password for authentification in "login:password" form, e. g. `druid.emitter.http.basicAuthentication=admin:adminpassword`|not specified = no authentification| | |||
| |`druid.emitter.http.flushTimeOut`|The timeout after which an event should be sent to the endpoint, even if internal buffers are not filled, in milliseconds.|not specified = no timeout| | |||
| |`druid.emitter.http.batchingStrategy`|The strategy of how the batch is formatted. "ARRAY" means `[event1,event2]`, "NEWLINES" means `event1\nevent2`, ONLY_EVENTS means `event1event2`.|ARRAY| | |||
| |`druid.emitter.http.maxBatchSize`|The maximum batch size, in bytes.|5191680 (i. e. 5 MB)| | |||
| |`druid.emitter.http.maxBatchSize`|The maximum batch size, in bytes.|the minimum of (10% of JVM heap size divided by 50) or (5191680 (i. e. 5 MB))| | |||
There was a problem hiding this comment.
In memory constrained settings I think it would be better to limit the batchQueueSizeLimit rather than the maxBatchSize. It's because very small batches might be an inefficient way to transfer metrics to the http server. We might want to limit both though. It's common to set Xmx to something small (like 64MB) for middleManagers, and in that case, probably 2 batches in the queue * 2.5 MB per batch makes sense.
There was a problem hiding this comment.
adjusted this to lower the queue limit first, while also keeping a minimum of 2 queue items
jon-wei
force-pushed
the
emitter_memory
branch
from
2b0fe96
to
7eaac35
Compare
jon-wei
added a commit
to implydata/druid-public
that referenced
this pull request
Mar 12, 2018
* More memory limiting for HttpPostEmitter * Less aggressive large events test * Fix tests * Restrict batch queue size first, keep minimum of 2 queue items
gianm
pushed a commit
to gianm/druid
that referenced
this pull request
Apr 9, 2018
* More memory limiting for HttpPostEmitter * Less aggressive large events test * Fix tests * Restrict batch queue size first, keep minimum of 2 queue items
fjy
pushed a commit
that referenced
this pull request
Apr 10, 2018
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adjusts the default
maxBatchSizefor the HttpPostEmitter such that the default configuration (nomaxBatchSizeorbatchQueueSizeLimitspecified) will limit the product ofmaxBatchSize*batchQueueSizeLimitto 10% of the available JVM heap.This can be useful since not all Druid nodes will be provisioned with the same amount of memory. In particular, the middle manager is often provisioned with a much smaller heap than the other nodes, which can result in OOM errors from buffered events when using the default values cluster-wide.
This PR also adjusts the large event handling such that they increment and decrement the
approximateBuffersToEmitCountcounter. Previously it was theoretically possible for the large event queue to keep growing past the configured queue limit.