Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merges might not be picked up when they are ready #5779

Closed
s1monw opened this Issue Apr 11, 2014 · 0 comments

Comments

Projects
None yet
1 participant
@s1monw
Copy link
Contributor

s1monw commented Apr 11, 2014

We currently play some tricks to make sure merges are not picked up on indexing threads such that the threadpool that is used for indexing is not consumed by merges. Yet, there are a couple of problems that prevent merges to be picked up since we are checking if the IW has pending merges but that datastructure might not be picked up due to a problem in the way we try to prevent the merges to happen. There is also a problem where we don't refresh a reader due to merges where we could / should do that. This does NOT have an impact on correctness but it can have an impact on the amount of segments that are used for searching and for resolving versions to. Both have performance impacts if the number of segments is large.

@s1monw s1monw added bug labels Apr 11, 2014

@s1monw s1monw removed the v1.0.3 label Apr 11, 2014

@s1monw s1monw closed this in 5d611a9 Apr 12, 2014

s1monw added a commit that referenced this issue Apr 12, 2014

Ensure pending merges are updated on segment flushes
Due to the default of `async_merge` to `true` we never run
the merge policy on a segment flush which prevented the
pending merges from being updated and that caused actual
pending merges not to contribute to the merge decision.

This commit also removes the `index.async.merge` setting is actually
misleading since we take care of merges not being excecuted on the
indexing threads  on a different level (the merge scheduler) since 1.1.

This commit also adds an additional check when to run a refresh
since soely relying on the dirty flag might leave merges un-refreshed
which can cause search slowdowns and higher memory consumption.

Closes #5779

s1monw added a commit that referenced this issue Apr 12, 2014

Ensure pending merges are updated on segment flushes
Due to the default of `async_merge` to `true` we never run
the merge policy on a segment flush which prevented the
pending merges from being updated and that caused actual
pending merges not to contribute to the merge decision.

This commit also removes the `index.async.merge` setting is actually
misleading since we take care of merges not being excecuted on the
indexing threads  on a different level (the merge scheduler) since 1.1.

This commit also adds an additional check when to run a refresh
since soely relying on the dirty flag might leave merges un-refreshed
which can cause search slowdowns and higher memory consumption.

Closes #5779

mute pushed a commit to mute/elasticsearch that referenced this issue Jul 29, 2015

Ensure pending merges are updated on segment flushes
Due to the default of `async_merge` to `true` we never run
the merge policy on a segment flush which prevented the
pending merges from being updated and that caused actual
pending merges not to contribute to the merge decision.

This commit also removes the `index.async.merge` setting is actually
misleading since we take care of merges not being excecuted on the
indexing threads  on a different level (the merge scheduler) since 1.1.

This commit also adds an additional check when to run a refresh
since soely relying on the dirty flag might leave merges un-refreshed
which can cause search slowdowns and higher memory consumption.

Closes elastic#5779
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.