Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Internal: Optimize with `force=true` can sidestep max segments to merge at once used by delegate #7904

Closed
rjernst opened this Issue Sep 26, 2014 · 1 comment

Comments

Projects
None yet
3 participants
@rjernst
Copy link
Member

commented Sep 26, 2014

The implementation of "forcing" optimize right now will cause the ElasticsearchMergePolicy to return one giant OneMerge if force==true. However, this can cause IO issues. The existing MP impls chain merging through "cascading", so that no OneMerge merges more than some X segments (e.g. X = 30 for TieredMP). Forcing should do the same...

@mikemccand

This comment has been minimized.

Copy link
Contributor

commented Sep 27, 2014

This is quite nasty: if a shard has many segments, this can suck up lots of RAM, file descriptors, take much longer to run than if we let the merge policy do separate merges ... I wonder how often users are "forcing" their optimize.

rjernst added a commit to rjernst/elasticsearch that referenced this issue Sep 29, 2014

Fix optimize behavior with 'force' and 'flush' flags.
This does the following:
* Make 'force' flag only build a merge if the delegate MP returned no merges
* Add async handling for 'flush' when 'waitForMerges' is false
* Remove flush at the beginning of optimize.  This is something the user can
  do if they wish, before calling optimize.

closes elastic#7886
closes elastic#7904

@rjernst rjernst closed this in 37b294a Sep 29, 2014

rjernst added a commit that referenced this issue Sep 29, 2014

Fix optimize behavior with 'force' and 'flush' flags.
This does the following:
* Make 'force' flag only build a merge if the delegate MP returned no merges
* Add async handling for 'flush' when 'waitForMerges' is false
* Remove flush at the beginning of optimize.  This is something the user can
  do if they wish, before calling optimize.

closes #7886
closes #7904
closes #7920

rjernst added a commit that referenced this issue Sep 30, 2014

Fix optimize behavior with 'force' and 'flush' flags.
This does the following:
* Make 'force' flag only build a merge if the delegate MP returned no merges
* Add async handling for 'flush' when 'waitForMerges' is false
* Remove flush at the beginning of optimize.  This is something the user can
  do if they wish, before calling optimize.

closes #7886
closes #7904
closes #7920

@rjernst rjernst added v1.4.0.Beta1 and removed v1.5.0 labels Sep 30, 2014

@clintongormley clintongormley changed the title Optimize with `force=true` can sidestep max segments to merge at once used by delegate Internal: Optimize with `force=true` can sidestep max segments to merge at once used by delegate Oct 1, 2014

mute pushed a commit to mute/elasticsearch that referenced this issue Jul 29, 2015

Fix optimize behavior with 'force' and 'flush' flags.
This does the following:
* Make 'force' flag only build a merge if the delegate MP returned no merges
* Add async handling for 'flush' when 'waitForMerges' is false
* Remove flush at the beginning of optimize.  This is something the user can
  do if they wish, before calling optimize.

closes elastic#7886
closes elastic#7904
closes elastic#7920
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.