Snapshot and restore and shrink do not play well together

[nik9000]

The simplest way to reproduce locally is with gradle run and then:

POST /test/doc
{ "test": "test" }


PUT /test/_settings
{
  "settings": {
    "index.routing.allocation.require._name": "shrink_node_name", 
    "index.blocks.write": true 
  }
}

POST /test/_shrink/shrunk

PUT /_snapshot/test_repo
{
  "type": "fs",
  "settings": {
    "compress": true,
    "location": "/Users/manybubbles/Workspaces/Elasticsearch/master/elasticsearch/distribution/build/cluster/shared/repo/test_repo"
  }
}

POST /_snapshot/test_repo/test_snapshot?wait_for_completion=true

POST /_snapshot/test_repo/test_snapshot/_restore?wait_for_completion=true
{
  "indices": ["shrunk"]
}

Kibana will give you "socket hang up" error because Elasticsearch has tripped an assertion and killed itself:

[elasticsearch] java.lang.AssertionError: all settings must have been upgraded before
[elasticsearch]         at org.elasticsearch.cluster.metadata.MetaDataIndexUpgradeService.upgradeIndexMetaData(MetaDataIndexUpgradeService.java:77) ~[elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]

If you run without -ea then the shrunken index won't allocation properly because of the index.routing.allocation.initial_recovery setting. That setting cannot be cleared.

[nik9000]

If you run without -ea then the shrunken index won't allocation properly because of the index.routing.allocation.initial_recovery setting. That setting cannot be cleared.

Clarification: if you run without -ea it won't recover properly unless the old node is around. But that is still a problem because then you can't snapshot in one cluster and restore to another. Or you can't restore after that node has been decommissioned.

[nik9000]

abeyad self-assigned this 3 minutes ago

Good luck!