Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

auto_expand_replicas: [0-all] can cause data loss when nodes are removed #934

Closed
clintongormley opened this issue May 16, 2011 · 1 comment

Comments

Projects
None yet
2 participants
@clintongormley
Copy link
Member

commented May 16, 2011

Hiya - there is a bug with auto_expand_replicas: [0-all] in v 0.16.1 which causes loss of all data in that index.

To replicate:

  • start two nodes
  • run the script below
  • count for index bar : 3
  • kill the node that holds the primary shard for index bar
  • count for index bar: 0

If you change auto expand to [1-all] then data is not lost.

curl -XDELETE 'http://127.0.0.1:9200/bar,foo/?pretty=1'

curl -XPUT 'http://127.0.0.1:9200/foo/?pretty=1'  -d '
{
   "settings" : {
      "number_of_replicas" : 0,
      "number_of_shards" : 1
   }
}
'

curl -XPUT 'http://127.0.0.1:9200/bar/?pretty=1'  -d '
{
   "settings" : {
      "index" : {
         "number_of_replicas" : 0,
         "number_of_shards" : 1
      }
   }
}
'


curl -XGET 'http://127.0.0.1:9200/_cluster/health/bar?pretty=1&wait_for_status=green' 


curl -XPOST 'http://127.0.0.1:9200/_bulk?pretty=1'  -d '
{"index" : {"_index" : "bar", "_type" : "name"}}
{"tokens" : ["stuart", "watt"], "context" : "/2850246/all", "rank" : 1}
{"index" : {"_index" : "bar", "_type" : "name"}}
{"tokens" : ["stuart", "watt"], "context" : "/2850246/jpnw/all", "rank" : 1}
{"index" : {"_index" : "bar", "_type" : "name"}}
{"tokens" : ["stuart", "watt"], "context" : "/2850246/jpnw_pres/all", "rank" : 1}
{"index" : {"_index" : "bar", "_type" : "name"}}
'

curl -XPOST 'http://127.0.0.1:9200/bar/_refresh?pretty=1' 

curl -XPUT 'http://127.0.0.1:9200/bar/_settings?pretty=1'  -d '
{
   "index" : {
      "auto_expand_replicas" : "0-all"
   }
}
'

curl -XGET 'http://127.0.0.1:9200/_cluster/health/bar?pretty=1&wait_for_status=green' 


curl -XGET 'http://127.0.0.1:9200/bar/_count?pretty=1'  -d '
{
   "match_all" : {}
}
'
@clintongormley

This comment has been minimized.

Copy link
Member Author

commented May 16, 2011

Note: I create two indices because the problem only shows up with both indices present.

@kimchy kimchy closed this in 518488b May 16, 2011

ofavre pushed a commit to yakaz/elasticsearch that referenced this issue Jul 18, 2011

auto_expand_replicas: [0-auto] can cause data loss when nodes are rem…
…oved, closes elastic#934.

This is caused because of a race condition between when to handle the removed node and move a replica to a primary mode, and when to remove the replica because of the 0-auto setting.

mute pushed a commit to mute/elasticsearch that referenced this issue Jul 29, 2015

auto_expand_replicas: [0-auto] can cause data loss when nodes are rem…
…oved, closes elastic#934.

This is caused because of a race condition between when to handle the removed node and move a replica to a primary mode, and when to remove the replica because of the 0-auto setting.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.