Skip to content

Commit

Permalink
Add note on the danger of closed indices (#61332)
Browse files Browse the repository at this point in the history
Prior to #33888 it is dangerous to keep closed indices in your cluster
long-term: Elasticsearch does not maintain their shard copies so they
tend to get lost as the cluster migrates to new nodes. This risk isn't
documented today. This commit addresses that gap.
  • Loading branch information
DaveCTurner committed Aug 20, 2020
1 parent fa39e7d commit 37825c3
Show file tree
Hide file tree
Showing 4 changed files with 31 additions and 12 deletions.
24 changes: 22 additions & 2 deletions docs/reference/indices/open-close.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,24 @@ for maintaining its metadata), and is blocked for read/write operations.
A closed index can be opened which will then go through the normal
recovery process.

[WARNING]
====
To reduce the risk of data loss, avoid keeping closed indices in your cluster
for long periods of time. When a node leaves your cluster, {es} does not
rebuild the lost copies of the shards in any closed indices. As you replace the
nodes in your cluster over time you will gradually lose the shard data for any
closed indices.
In managed environments such as https://www.elastic.co/cloud[Elastic Cloud]
nodes may be automatically replaced at any time, which will result in the loss
of data held in closed indices. You should not close any indices if your
cluster is running in such a a managed environment.
If you want to reduce the overhead of an index while keeping it available for
occasional searches, <<frozen-indices,freeze>> the index instead. If you want
to store an index outside of the cluster, use a <<modules-snapshots,snapshot>>.
====

The REST endpoint is `/{index}/_close` and `/{index}/_open`. For
example:

Expand All @@ -30,8 +48,10 @@ Identifying indices via wildcards or `_all` can be disabled by setting the
`action.destructive_requires_name` flag in the config file to `true`.
This setting can also be changed via the cluster update settings api.

Closed indices consume a significant amount of disk-space which can cause problems in managed environments. Closing indices can be disabled via the cluster settings
API by setting `cluster.indices.close.enable` to `false`. The default is `true`.
Closed indices consume a significant amount of disk-space which can cause
problems in managed environments. Closed indices are enabled by default. You can
disable closed indices by setting `cluster.indices.close.enable` to `false`
using the <<cluster-update-settings,cluster settings API>>.

IMPORTANT: Closed indices are ignored by many APIs. For instance, the shards of
a closed index are not included in the output of the <<cat-shards>> API.
Expand Down
8 changes: 3 additions & 5 deletions docs/reference/setup/install.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,9 @@

[float]
=== Hosted Elasticsearch
Elasticsearch can be run on your own hardware or using our hosted
Elasticsearch Service on https://www.elastic.co/cloud[Elastic Cloud], which is
available on AWS and GCP. You can
https://www.elastic.co/cloud/elasticsearch-service/signup[try out the hosted service] for free.
{es} can be run on your own hardware or using our hosted {ess} on
https://www.elastic.co/cloud[{ecloud}], which is available on AWS, GCP and
Azure. You can {ess-trial}[try out the hosted service] for free.

[float]
=== Installing Elasticsearch Yourself
Expand Down Expand Up @@ -76,4 +75,3 @@ include::install/rpm.asciidoc[]
include::install/windows.asciidoc[]

include::install/docker.asciidoc[]

Original file line number Diff line number Diff line change
Expand Up @@ -84,8 +84,9 @@ protected AcknowledgedResponse newResponse() {
protected void doExecute(Task task, CloseIndexRequest request, ActionListener<AcknowledgedResponse> listener) {
destructiveOperations.failDestructive(request.indices());
if (closeIndexEnabled == false) {
throw new IllegalStateException("closing indices is disabled - set [" + CLUSTER_INDICES_CLOSE_ENABLE_SETTING.getKey() +
": true] to enable it. NOTE: closed indices still consume a significant amount of diskspace");
throw new IllegalStateException("closing indices is forbidden since data in closed indices may be lost during migrations and " +
"upgrades, and they also consume significant disk space; set [" + CLUSTER_INDICES_CLOSE_ENABLE_SETTING.getKey()
+ ": true] to permit indices to be closed");
}
super.doExecute(task, request, listener);
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -49,9 +49,9 @@ public void testCloseAllRequiresName() {

IllegalStateException illegalStateException = expectThrows(IllegalStateException.class,
() -> client().admin().indices().prepareClose("test_no_close").get());
assertEquals(illegalStateException.getMessage(),
"closing indices is disabled - set [cluster.indices.close.enable: true] to enable it. NOTE: closed indices still " +
"consume a significant amount of diskspace");
assertEquals(illegalStateException.getMessage(), "closing indices is forbidden since data in closed indices may be lost during " +
"migrations and upgrades, and they also consume significant disk space; set [cluster.indices.close.enable: true] to " +
"permit indices to be closed");
}

private void assertIndexIsClosed(String... indices) {
Expand Down

0 comments on commit 37825c3

Please sign in to comment.