[DOCS] add warning for read-write indices in force merge documentation #28869

mueedc-zz · 2018-03-01T16:31:22Z

This PR updates force merge documentation:

Add warning message to specify force merge API calls should only be used against read-only indices

elasticmachine · 2018-03-01T16:31:23Z

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

elasticmachine · 2018-03-01T16:31:23Z

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

javanna · 2018-03-02T09:59:33Z

@ppf2 would you like to have a look at this?

jpountz

Looks good to me overall, I left some minor comments.

jpountz · 2018-03-15T14:55:04Z

docs/reference/indices/forcemerge.asciidoc

+Force merge should only be called against *read-only indices*. Running 
+force merge against a read-write index can cause very large segments to be produced 
+(>5Gb per segment), and the merge policy will never consider it for merging again until 
+it has 75% of deleted docs. This can cause very large segments to remain in the shards.


I don't think this 75% is correct, it would be shard_size/5GB but I think it's fine to say "until it mostly consists of deleted docs".

jpountz · 2018-03-15T14:55:55Z

docs/reference/indices/forcemerge.asciidoc

@@ -10,6 +10,12 @@ This call will block until the merge is complete. If the http connection is
 lost, the request will continue in the background, and any new requests will
 block until the previous force merge is complete.

+[WARNING]
+Force merge should only be called against *read-only indices*. Running 


Just do WARNING: Force merge... and our documentation processor will make a nice warning block out of this paragraph.

mueedc-zz · 2018-03-15T15:48:23Z

@jpountz your requested changes have been added and the commit was rebased/squashed back into one

ppf2 · 2018-03-15T18:40:53Z

lgtm thanks all!

elasticmachine · 2018-04-24T08:39:45Z

Pinging @elastic/es-core-infra

gondalez · 2018-04-30T03:20:44Z

Hi and thanks for this very useful addition to the docs 😌

Do you think it would be worth adding an accompanying tip around re-running force merge after altering a lot of documents?

I'll attempt to explain by example:

Our use case is analytics on time series data. We have one index per month where new data is only written to the newer indices. We force merge the older indices for optimal query times.

Every now and again we need to add a new attribute to all of the docs and populate using scripted update by query or similar.
At the end of the process we have to manually force merge the older indices to purge the very many deleted docs and reclaim space. If we do not manually force merge the deleted docs remain.

Having read this issue (thanks again!) I now know that it is because our old already-force-merged indices are not eligible for cleanup due to them having been force merged in the past.

Adding a note might help future travellers. wdyt?

mueedc-zz · 2018-04-30T19:08:08Z

@jpountz I can add a tip for re-running force merge after altering many documents as per @gondalez comment. let me know your thoughts

javanna · 2018-05-07T12:32:33Z

@jpountz can you have another look please?

jpountz · 2018-05-09T16:50:10Z

@gondalez I have seen too many issues when force-merge was run too often, and basically none when it wasn't run when it should so I'd rather keep the messaging clear for now and only say that force-merge should only run against read-only indices.

#28869)

* master: Upgrade to Lucene-7.4-snapshot-6705632810 (#30519) add version compatibility from 6.4.0 after backport, see #30319 (#30390) Security: Simplify security index listeners (#30466) Add proper longitude validation in geo_polygon_query (#30497) Remove Discovery.AckListener.onTimeout() (#30514) Build: move generated-resources to build (#30366) Reindex: Fold "with all deps" project into reindex (#30154) Isolate REST client single host tests (#30504) Solve Gradle deprecation warnings around shadowJar (#30483) SAML: Process only signed data (#30420) Remove BWC repository test (#30500) Build: Remove xpack specific run task (#30487) AwaitsFix IntegTestZipClientYamlTestSuiteIT#indices.split tests LLClient: Add setJsonEntity (#30447) Expose CommonStatsFlags directly in IndicesStatsRequest. (#30163) Silence IndexUpgradeIT test failures. (#30430) Bump Gradle heap to 1792m (#30484) [docs] add warning for read-write indices in force merge documentation (#28869) Avoid deadlocks in cache (#30461) Test: remove hardcoded list of unconfigured ciphers (#30367) mute SplitIndexIT due to #30416 Docs: Test examples that recreate lang analyzers (#29535) BulkProcessor to retry based on status code (#29329) Add GET Repository High Level REST API (#30362) add a comment explaining the need for RetryOnReplicaException on missing mappings Add `coordinating_only` node selector (#30313) Stop forking groovyc (#30471) Avoid setting connection request timeout (#30384) Use date format in `date_range` mapping before fallback to default (#29310) Watcher: Increase HttpClient parallel sent requests (#30130) # Conflicts: # x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/LocalStateCompositeXPackPlugin.java

* 6.x: Upgrade to Lucene-7.4-snapshot-6705632810 (#30519) Remove Discovery.AckListener.onTimeout() (#30514) Build: move generated-resources to build (#30366) Reindex: Fold "with all deps" project into reindex (#30154) Isolate REST client single host tests (#30504) Remove BWC repository test (#30500) Build: Remove xpack specific run task (#30487) AwaitsFix IntegTestZipClientYamlTestSuiteIT#indices.split tests LLClient: Add setJsonEntity (#30447) [docs] add warning for read-write indices in force merge documentation (#28869) Avoid deadlocks in cache (#30461) BulkProcessor to retry based on status code (#29329) Avoid setting connection request timeout (#30384) Test: remove hardcoded list of unconfigured ciphers (#30367) Add GET Repository High Level REST API (#30362) mute SplitIndexIT due to #30416 Docs: Test examples that recreate lang analyzers (#29535) add a comment explaining the need for RetryOnReplicaException on missing mappings Pass the task to broadcast actions (#29672) Stop forking groovyc (#30471) Add `coordinating_only` node selector (#30313) Fix accidental error in changelog Use date format in `date_range` mapping before fallback to default (#29310) Watcher: Increase HttpClient parallel sent requests (#30130) [Security][Tests] Azeri(Turkish) locale tripps opensaml dependency

* es/ccr: (78 commits) Upgrade to Lucene-7.4-snapshot-6705632810 (elastic#30519) add version compatibility from 6.4.0 after backport, see elastic#30319 (elastic#30390) Security: Simplify security index listeners (elastic#30466) Add proper longitude validation in geo_polygon_query (elastic#30497) Remove Discovery.AckListener.onTimeout() (elastic#30514) Build: move generated-resources to build (elastic#30366) Reindex: Fold "with all deps" project into reindex (elastic#30154) Isolate REST client single host tests (elastic#30504) Solve Gradle deprecation warnings around shadowJar (elastic#30483) SAML: Process only signed data (elastic#30420) Remove BWC repository test (elastic#30500) Build: Remove xpack specific run task (elastic#30487) AwaitsFix IntegTestZipClientYamlTestSuiteIT#indices.split tests Enable soft-deletes in v6.4 LLClient: Add setJsonEntity (elastic#30447) [CCR] Read changes from Lucene instead of translog (elastic#30120) Expose CommonStatsFlags directly in IndicesStatsRequest. (elastic#30163) Silence IndexUpgradeIT test failures. (elastic#30430) Bump Gradle heap to 1792m (elastic#30484) [docs] add warning for read-write indices in force merge documentation (elastic#28869) ...

silashansen · 2018-08-21T08:44:53Z

I think some information on what to do if a force merge was already issued would be welcomed.

How do you get the index back to a state where the normal cleanup policies will apply?

mueedc-zz changed the title ~~[docs] add warning for read-write indices in force merge documentation~~ [DOCS] add warning for read-write indices in force merge documentation Mar 1, 2018

javanna requested a review from ppf2 March 2, 2018 09:59

javanna added the >docs General docs changes label Mar 2, 2018

jpountz requested changes Mar 15, 2018

View reviewed changes

[docs] add warning for read-write indices in force merge documentation

fc295f0

mueedc-zz force-pushed the master branch from 5134c35 to fc295f0 Compare March 15, 2018 15:47

colings86 added the :Data Management/Indices APIs APIs to create and manage indices and templates label Apr 24, 2018

javanna added the review label May 7, 2018

jpountz approved these changes May 9, 2018

View reviewed changes

jpountz added :Data Management/Indices APIs APIs to create and manage indices and templates >docs General docs changes and removed :Data Management/Indices APIs APIs to create and manage indices and templates >docs General docs changes labels May 9, 2018

jpountz merged commit bf141a3 into elastic:master May 9, 2018

jpountz pushed a commit that referenced this pull request May 9, 2018

[docs] add warning for read-write indices in force merge documentation (

5a12c11

#28869)

jpountz pushed a commit that referenced this pull request May 9, 2018

[docs] add warning for read-write indices in force merge documentation (

2f6188a

#28869)

This was referenced Apr 28, 2019

Allow force-merge on read-only indices #41624

Closed

Recommend force-merging after stopping indexing but before marking the index read-only #41687

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DOCS] add warning for read-write indices in force merge documentation #28869

[DOCS] add warning for read-write indices in force merge documentation #28869

mueedc-zz commented Mar 1, 2018 •

edited by javanna

elasticmachine commented Mar 1, 2018

elasticmachine commented Mar 1, 2018

javanna commented Mar 2, 2018

jpountz left a comment

jpountz Mar 15, 2018

jpountz Mar 15, 2018

mueedc-zz commented Mar 15, 2018

ppf2 commented Mar 15, 2018

elasticmachine commented Apr 24, 2018

gondalez commented Apr 30, 2018

mueedc-zz commented Apr 30, 2018

javanna commented May 7, 2018

jpountz commented May 9, 2018

silashansen commented Aug 21, 2018

[DOCS] add warning for read-write indices in force merge documentation #28869

[DOCS] add warning for read-write indices in force merge documentation #28869

Conversation

mueedc-zz commented Mar 1, 2018 • edited by javanna

elasticmachine commented Mar 1, 2018

elasticmachine commented Mar 1, 2018

javanna commented Mar 2, 2018

jpountz left a comment

Choose a reason for hiding this comment

jpountz Mar 15, 2018

Choose a reason for hiding this comment

jpountz Mar 15, 2018

Choose a reason for hiding this comment

mueedc-zz commented Mar 15, 2018

ppf2 commented Mar 15, 2018

elasticmachine commented Apr 24, 2018

gondalez commented Apr 30, 2018

mueedc-zz commented Apr 30, 2018

javanna commented May 7, 2018

jpountz commented May 9, 2018

silashansen commented Aug 21, 2018

mueedc-zz commented Mar 1, 2018 •

edited by javanna