Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Internal: Optimize with force=true can sidestep max segments to merge at once used by delegate #7904

Closed
rjernst opened this issue Sep 26, 2014 · 1 comment
Closed

Internal: Optimize with force=true can sidestep max segments to merge at once used by delegate #7904

rjernst opened this issue Sep 26, 2014 · 1 comment
Assignees
@rjernst
Labels
>bug v1.4.0.Beta1 v2.0.0-beta1

Comments

@rjernst
Copy link
Member

rjernst commented Sep 26, 2014

The implementation of "forcing" optimize right now will cause the ElasticsearchMergePolicy to return one giant OneMerge if force==true. However, this can cause IO issues. The existing MP impls chain merging through "cascading", so that no OneMerge merges more than some X segments (e.g. X = 30 for TieredMP). Forcing should do the same...
@mikemccand
Copy link
Contributor

This is quite nasty: if a shard has many segments, this can suck up lots of RAM, file descriptors, take much longer to run than if we let the merge policy do separate merges ... I wonder how often users are "forcing" their optimize.

rjernst added a commit to rjernst/elasticsearch that referenced this issue Sep 29, 2014
@rjernst
Fix optimize behavior with 'force' and 'flush' flags.
4ce798d 
This does the following:
* Make 'force' flag only build a merge if the delegate MP returned no merges
* Add async handling for 'flush' when 'waitForMerges' is false
* Remove flush at the beginning of optimize.  This is something the user can
  do if they wish, before calling optimize.

closes elastic#7886
closes elastic#7904
@rjernst rjernst mentioned this issue Sep 29, 2014
Closed
rjernst added a commit that referenced this issue Sep 29, 2014
@rjernst
Fix optimize behavior with 'force' and 'flush' flags.
73b9205 
This does the following:
* Make 'force' flag only build a merge if the delegate MP returned no merges
* Add async handling for 'flush' when 'waitForMerges' is false
* Remove flush at the beginning of optimize.  This is something the user can
  do if they wish, before calling optimize.

closes #7886
closes #7904
closes #7920
rjernst added a commit that referenced this issue Sep 30, 2014
@rjernst
Fix optimize behavior with 'force' and 'flush' flags.
90db591 
This does the following:
* Make 'force' flag only build a merge if the delegate MP returned no merges
* Add async handling for 'flush' when 'waitForMerges' is false
* Remove flush at the beginning of optimize.  This is something the user can
  do if they wish, before calling optimize.

closes #7886
closes #7904
closes #7920
@clintongormley clintongormley changed the title Optimize with force=true can sidestep max segments to merge at once used by delegate Internal: Optimize with force=true can sidestep max segments to merge at once used by delegate Oct 1, 2014
mute pushed a commit to mute/elasticsearch that referenced this issue Jul 29, 2015
@rjernst
Fix optimize behavior with 'force' and 'flush' flags.
2871b03 
This does the following:
* Make 'force' flag only build a merge if the delegate MP returned no merges
* Add async handling for 'flush' when 'waitForMerges' is false
* Remove flush at the beginning of optimize.  This is something the user can
  do if they wish, before calling optimize.

closes elastic#7886
closes elastic#7904
closes elastic#7920
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rjernst rjernst

Labels
>bug v1.4.0.Beta1 v2.0.0-beta1
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix optimize behavior with 'force' and 'flush' flags. rjernst/elasticsearch
3 participants
@rjernst @mikemccand @s1monw