Reduce merging in PersistedClusterStateService #79793
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When writing the cluster state index we flush a segment every 2000 docs or so, which sometimes triggers merging in the middle of the write process. This merging is often unnecessary since many of the segments being merged would have ended up containing no live docs at the end of the process and hence could have just been deleted. With this commit we adjust the merge policy to be much more relaxed about merging, permitting up to 100 segments per tier, since we only read this index very rarely and not on any hot paths. We also disable merging completely during the write process, checking just before commit to see if any merging should be done. Relates elastic#77466
DaveCTurner
added
>bug
:Distributed/Cluster Coordination
|
Pinging @elastic/es-distributed (Team:Distributed) |
97 tasks
original-brownbear
approved these changes
Oct 26, 2021
DaveCTurner
added
the
auto-backport-and-merge
💔 Backport failed
You can use sqren/backport to manually backport by running |
DaveCTurner
added a commit
that referenced
this pull request
Oct 26, 2021
When writing the cluster state index we flush a segment every 2000 docs or so, which sometimes triggers merging in the middle of the write process. This merging is often unnecessary since many of the segments being merged would have ended up containing no live docs at the end of the process and hence could have just been deleted. With this commit we adjust the merge policy to be much more relaxed about merging, permitting up to 100 segments per tier, since we only read this index very rarely and not on any hot paths. We also disable merging completely during the write process, checking just before commit to see if any merging should be done. Relates #77466
weizijun
added a commit
to weizijun/elasticsearch
that referenced
this pull request
Oct 26, 2021
* upstream/master: (209 commits) Enforce license expiration (elastic#79671) TSDB: Automatically add timestamp mapper (elastic#79136) [DOCS] `_id` is required for bulk API's `update` action (elastic#79774) EQL: Add optional fields and limit joining keys on non-null values only (elastic#79677) [DOCS] Document range enrich policy (elastic#79607) [DOCS] Fix typos in 8.0 security migration (elastic#79802) Allow listing older repositories (elastic#78244) [ML] track inference model feature usage per node (elastic#79752) Remove IncrementalClusterStateWriter & related code (elastic#79738) Reuse previous indices lookup when possible (elastic#79004) Reduce merging in PersistedClusterStateService (elastic#79793) SQL: Adjust JDBC docs to use milliseconds for timeouts (elastic#79628) Remove endpoint for freezing indices (elastic#78918) [ML] add timeout parameter for DELETE trained_models API (elastic#79739) [ML] wait for .ml-state-write alias to be readable (elastic#79731) [Docs] Update edgengram-tokenizer.asciidoc (elastic#79577) [Test][Transform] fix UpdateTransformActionRequestTests failure (elastic#79787) Limit CS Update Task Description Size (elastic#79443) Apply the reader wrapper on can_match source (elastic#78988) [DOCS] Adds new transform limitation item and a note to the tutorial (elastic#79479) ... # Conflicts: # server/src/main/java/org/elasticsearch/index/IndexMode.java # server/src/test/java/org/elasticsearch/index/TimeSeriesModeTests.java
lockewritesdocs
pushed a commit
to lockewritesdocs/elasticsearch
that referenced
this pull request
Oct 28, 2021
When writing the cluster state index we flush a segment every 2000 docs or so, which sometimes triggers merging in the middle of the write process. This merging is often unnecessary since many of the segments being merged would have ended up containing no live docs at the end of the process and hence could have just been deleted. With this commit we adjust the merge policy to be much more relaxed about merging, permitting up to 100 segments per tier, since we only read this index very rarely and not on any hot paths. We also disable merging completely during the write process, checking just before commit to see if any merging should be done. Relates elastic#77466
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When writing the cluster state index we flush a segment every 2000 docs
or so, which sometimes triggers merging in the middle of the write
process. This merging is often unnecessary since many of the segments
being merged would have ended up containing no live docs at the end of
the process and hence could have just been deleted.
With this commit we adjust the merge policy to be much more relaxed
about merging, permitting up to 100 segments per tier, since we only
read this index very rarely and not on any hot paths. We also disable
merging completely during the write process, checking just before commit
to see if any merging should be done.
Relates #77466