conflicts with existing mapping in other types:

[makeyang]

Elasticsearch version:
2.1
Plugins installed: []
delete-by-query
elasticsearch-analysis-ik
repository-hdfs
JVM version:
8u60
OS version:
CentOS release 6.6 (Final)
Description of the problem including expected versus actual behavior:
server keep throw below exceptions:

[WARN ][cluster.action.shard     ] [dm_xxx] [form_index][8] received shard failed for [form_index][8], node[Yz0FhSvfS2C9Ka_fUhywKQ], [P], v[104], s[INITIALIZING], a[id=9iRqwiV8Qpu2_uZc4nW_Iw], unassigned_info[[reason=ALLOCATION_FAILED], at[2016-12-30T08:20:32.956Z], details[failed to update mappings, failure IllegalArgumentException[Mapper for [lockUser] conflicts with existing mapping in other types:
[mapper [lockUser] has different [index] values, mapper [lockUser] has different [doc_values] values, cannot change from disabled to enabled, mapper [lockUser] has different [analyzer]]]]], indexUUID [Ww4WaOiPS2Oy09E_MAWPHA], message [failed recovery], failure [IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to recover from translog]; nested: EngineException[failed to recover from translog]; nested: TypeMissingException[type[[formData, trying to auto create mapping, but dynamic mapping is disabled]] missing]; ]
[form_index][[form_index][8]] IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to recover from translog]; nested: EngineException[failed to recover from translog]; nested: TypeMissingException[type[[formData, trying to auto create mapping, but dynamic mapping is disabled]] missing];
        at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:254)
        at org.elasticsearch.index.shard.StoreRecoveryService.access$100(StoreRecoveryService.java:56)
        at org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:129)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: [form_index][[form_index][8]] EngineCreationFailureException[failed to recover from translog]; nested: EngineException[failed to recover from translog]; nested: TypeMissingException[type[[formData, trying to auto create mapping, but dynamic mapping is disabled]] missing];
        at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:178)
        at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25)
        at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1437)
        at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1421)
        at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:920)
        at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:892)
        at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:245)
        ... 5 more
Caused by: [form_index][[form_index][8]] EngineException[failed to recover from translog]; nested: TypeMissingException[type[[formData, trying to auto create mapping, but dynamic mapping is disabled]] missing];
        at org.elasticsearch.index.engine.InternalEngine.recoverFromTranslog(InternalEngine.java:254)
        at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:175)
        ... 11 more
Caused by: [form_index] TypeMissingException[type[[formData, trying to auto create mapping, but dynamic mapping is disabled]] missing]
        at org.elasticsearch.index.mapper.MapperService.documentMapperWithAutoCreate(MapperService.java:416)
        at org.elasticsearch.index.shard.TranslogRecoveryPerformer.docMapper(TranslogRecoveryPerformer.java:74)
        at org.elasticsearch.index.shard.TranslogRecoveryPerformer.performRecoveryOperation(TranslogRecoveryPerformer.java:163)
        at org.elasticsearch.index.engine.InternalEngine.recoverFromTranslog(InternalEngine.java:242)
        ... 12 more

Steps to reproduce:
I checked the code complete and it turns out it really hard to reproduce this issue:

1. when put/update mapping on master, the merge mapping action is execute in single thread and if any mapping conflict, it won't send mapping to data node but just throw exception
2. when put mappping below:

twitter' -d'
{  "settings": {    "index": {	  "unassigned.node_left.delayed_timeout": "5m",      "number_of_shards": "2",      "number_of_replicas": "0"    }  },
  "mappings": {    "tweet1": {      "properties": {        "message": {          "type": "string",		  "index":      "not_analyzed",		  "doc_values": true        }      }    }  }
} 

add some data and then kill one data node. then update mapping

{
    "properties": {        "message": {		  "type": "string","doc_values": false, "index":      "not_analyzed",    "norms": {"enabled": false}    }    }
}

and restart the killed node which has translog in disk. but the mapping will be merged sinnce doc value can changed from enable to disable.

3. when put mapping below:

curl -XPUT '192.168.200.191:9405/twitter' -d'
{  "settings": {    "index": {	  "unassigned.node_left.delayed_timeout": "5m",      "number_of_shards": "2",      "number_of_replicas": "0"    }  },
  "mappings": {    "tweet1": {      "properties": {        "message": {          "type": "string",		  "index":      "not_analyzed",		  "doc_values": false        }      }    }  }
}
'

add some data and then kill one data node. then update mapping

{
    "properties": {        "message": {		  "type": "string","doc_values": true, "index":      "not_analyzed",    "norms": {"enabled": false}    }    }
}

then merge exception will throw on master node.
so can anybody help to investigate on this please?

[makeyang]

just a thought:

1. when update mapping route to a master and before it is publishing to all data nodes, network partition happens
2. new master is elected and another update mapping route to new master.
3. this 2 mapping has conflict
4. when old master come back and it tried to rejoin the cluster, will this cause this exception?
   I'll try it tomorrow.

[makeyang]

I try this and it is still no reproduce. can anybody share anything?

[bleskes]

@makeyang can you share a bit more context? is this a cluster with dedicated master nodes? is this an old index (created with <=1.x code). Since this is an initializing primary, it seems something happened before - was the cluster restarted? is this a restore from a snapshot? any networking issues?

[makeyang]

@bleskes

1. no, it's not a cluster with dedicated master nodes. master&data nodes are integrated together with seperate gateway node. it's 3 data&master integrated nodes with 4 dedicated gateway nodes.
2. no, it's not an old index. the index was created & maintained in the same cluster and the cluster version is 2.1
3. we update conflict mapping by not knowing why and then we found out that some shards can't be allocated. then we restart some data&master node
4. then it keep throw exceptions.

I paste 3 data&master nodes log here:
https://github.com/makeyang/grocery/blob/master/15921.txt
https://github.com/makeyang/grocery/blob/master/8045.txt
https://github.com/makeyang/grocery/blob/master/8047.txt

[ywelsch]

This is a bug in the mapping code. I've got a simple reproduction on a single-node 2.1.0 cluster:

curl -XPUT localhost:9200/my_index

curl -XPUT localhost:9200/my_index/_mapping/type1 -d '
{
  "properties": {
    "account": {
      "type": "string",
      "index": "not_analyzed",
      "doc_values": true
    }
  }
}
'

curl -XPUT localhost:9200/my_index/_mapping/type2 -d '
{
  "properties": {
    "account": {
      "type": "string",
      "index": "not_analyzed",
      "doc_values": false
    }
  }
}
'

Then restart the node and see it fail recovering. If it does not fail the first time, restart a few more times, the failure depends on the order in which type mappings are applied, which is somewhat non-deterministic (hashmaps).

The issue is that the mapping ends up with conflicting values for the "account" doc-values field (which is not properly detected by ES 2.1.0):

{
    "my_index": {
        "mappings": {
            "type1": {
                "properties": {
                    "account": {
                        "index": "not_analyzed",
                        "type": "string"
                    }
                }
            },
            "type2": {
                "properties": {
                    "account": {
                        "doc_values": false,
                        "index": "not_analyzed",
                        "type": "string"
                    }
                }
            }
        }
    }
}

i.e. doc_values is enabled for the "account" field in type1 but disabled in type2, which is inconsistent. The validation only detects this based on the ordering in which type1 and type2 are applied when applying the mapping on the node.

ES 2.4.4 updates the mapping correctly and yields the following consistent mapping after applying the above steps:

{
    "my_index": {
        "mappings": {
            "type1": {
                "properties": {
                    "account": {
                        "doc_values": false,
                        "index": "not_analyzed",
                        "type": "string"
                    }
                }
            },
            "type2": {
                "properties": {
                    "account": {
                        "doc_values": false,
                        "index": "not_analyzed",
                        "type": "string"
                    }
                }
            }
        }
    }
}

How to fix this? You can make the mapping consistent again by disabling doc_values for the form_index/lockUser field for all types in that index:

In my above example, the mapping can be made consistent again by updating it as follows:

curl -XPUT localhost:9200/my_index/_mapping/type1?update_all_types=true -d '
{
  "properties": {
    "account": {
      "type": "string",
      "index": "not_analyzed",
      "doc_values": false
    }
  }
}
'

This command might fail the first time you execute it, as it runs again into the non-determinism issue. Just repeat the command a few times until you get {"acknowledged":true}.

Once the mapping has been fixed (and only thereafter), I recommend an upgrade to ES 2.4.4, preferably even to 5.x.

[makeyang]

@ywelsch
after dig more: i setup a cluster with one dedicated master and one dedicated data node. and below test will throw exception too:
curl -XPUT /my_index2 -d'
{ "settings" : { "index" : { "number_of_shards" : 1,"number_of_replicas" : 0 }}}
'

curl -XPUT /my_index2/_mapping/type1 -d '
{
"properties": {
"account": {
"type": "string"
}
}
}
'

curl -XPUT /my_index2/_mapping/type2 -d '
{ "properties": { "account": { "index": "not_analyzed", "type": "string" } }
}
'

and then I found that below issues are trying to fix mapping problem:
#15049
#15057
#17568

so basiclly we can close this issue.