Improve IndexingMemoryController a bit #13548
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- promptly push indexing buffer changes to IndexWriter, instead of waiting for next refresh/flush - don't wait for merges to finish before dropping a shards's indexing buffer to 512 KB - fix NPE if indices.memory.index_buffer_size is in node's settings with a bytes (not %) unit - add some more logger.debug
| * This setting is <b>not</b> realtime updateable. | ||
| * Index setting to control the initial index buffer size. NOTE: this setting is somewhat | ||
| * useless, since IndexingMemoryController will take over quickly and partition the | ||
| * indices.memory.index_buffer_size for this node across all shards. |
There was a problem hiding this comment.
do we want to remove it for 2.0 then?
There was a problem hiding this comment.
do we want to remove it for 2.0 then?
+1, but I'll open a separate issue.
There was a problem hiding this comment.
+1, but I'll open a separate issue.
I opened #13563 for this.
|
Thx mike. Left comments/questions here and there. |
…uffer size in node settings on node init took
|
Thanks @bleskes, I folded in the feedback! |
|
|
||
| this.interval = this.settings.getAsTime(SHARD_INACTIVE_INTERVAL_TIME_SETTING, TimeValue.timeValueSeconds(30)); | ||
|
|
||
| logger.debug("using indexing buffer size [{}], with {} [{}], {} [{}], {} [{}], {} [{}]", |
|
LGTM. Thx @mikemccand |
mikemccand
pushed a commit
that referenced
this pull request
Sep 15, 2015
Improve IndexingMemoryController a bit:
- promptly push indexing buffer changes to IndexWriter, instead of waiting for next refresh/flush
- don't wait for merges to finish before dropping a shards's indexing buffer to 512 KB once it's inactive
- fix NPE if indices.memory.index_buffer_size is in node's settings with a bytes (not %) unit
- add some more logger.debug
mikemccand
pushed a commit
that referenced
this pull request
Sep 15, 2015
Improve IndexingMemoryController a bit:
- promptly push indexing buffer changes to IndexWriter, instead of waiting for next refresh/flush
- don't wait for merges to finish before dropping a shards's indexing buffer to 512 KB once it's inactive
- fix NPE if indices.memory.index_buffer_size is in node's settings with a bytes (not %) unit
- add some more logger.debug
mikemccand
pushed a commit
that referenced
this pull request
Sep 15, 2015
Improve IndexingMemoryController a bit:
- promptly push indexing buffer changes to IndexWriter, instead of waiting for next refresh/flush
- don't wait for merges to finish before dropping a shards's indexing buffer to 512 KB once it's inactive
- fix NPE if indices.memory.index_buffer_size is in node's settings with a bytes (not %) unit
- add some more logger.debug
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I started by trying to explain the test failure in #13487, but failed so far, but then in the process found some other things to fix:
IndexingMemoryControllersets indexing buffer sizes for each shard, but then doesn't always push those changes down toIndexWriterindices.memory.index_buffer_sizein the node settings to a bytes (not %) value, you'll hit NPE on starting up the nodeI think we should separately fix these (this PR) ... but I'll keep trying to explain the original test failure.
I did add some more
logger.debugso maybe next failure reveals something.