Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conflicting mappings causes runaway shard failures on upgrade #11857

Closed
clintongormley opened this Issue Jun 24, 2015 · 2 comments

Comments

Projects
None yet
5 participants
@clintongormley
Copy link
Member

commented Jun 24, 2015

Start Elasticsearch 1.6.0, and run the following commands:

DELETE *

PUT bad
{
  "mappings": {
    "x": {
      "properties": {
        "foo": {
          "type": "string"
        }
      }
    },
    "y": {
      "properties": {
        "foo": {
          "type": "date"
        }
      }
    }
  }
}

PUT bad/x/1
{"foo": "bar"}

PUT bad/y/1
{"foo": "2015-01-01"}

POST _flush

Then shutdown and start ES compiled from master. The bad index fails to recover because of the conflicting mappings, and the failures just keep being repeated endlessly:

[2015-06-24 18:22:46,644][WARN ][indices.cluster          ] [Crossbones] [bad] failed to add mapping [y], source [{"y":{"properties":{"foo":{"t
ype":"date","format":"dateOptionalTime"}}}}]
java.lang.IllegalArgumentException: Mapper for [foo] conflicts with existing mapping in other types[mapper [foo] cannot be changed from type [s
tring] to [date]]
        at org.elasticsearch.index.mapper.FieldTypeLookup.checkCompatibility(FieldTypeLookup.java:117)
        at org.elasticsearch.index.mapper.MapperService.checkNewMappersCompatibility(MapperService.java:350)
        at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:305)
        at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:255)
        at org.elasticsearch.indices.cluster.IndicesClusterStateService.processMapping(IndicesClusterStateService.java:422)
        at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyMappings(IndicesClusterStateService.java:376)
        at org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:181)
        at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:484)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThr
eadPoolExecutor.java:209)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolE
xecutor.java:179)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)
[2015-06-24 18:22:46,701][WARN ][indices.cluster          ] [Crossbones] [[bad][0]] marking and sending shard failed due to [failed to update m
appings]
java.lang.IllegalArgumentException: Mapper for [foo] conflicts with existing mapping in other types[mapper [foo] cannot be changed from type [s
tring] to [date]]
        at org.elasticsearch.index.mapper.FieldTypeLookup.checkCompatibility(FieldTypeLookup.java:117)
        at org.elasticsearch.index.mapper.MapperService.checkNewMappersCompatibility(MapperService.java:350)
        at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:305)
        at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:255)
        at org.elasticsearch.indices.cluster.IndicesClusterStateService.processMapping(IndicesClusterStateService.java:422)
        at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyMappings(IndicesClusterStateService.java:376)
        at org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:181)
        at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:484)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThr
eadPoolExecutor.java:209)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolE
xecutor.java:179)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)
[2015-06-24 18:22:46,703][WARN ][cluster.action.shard     ] [Crossbones] [bad][0] received shard failed for [bad][0], node[_DucCUykRy6JHLfcchxVVg], [P], s[INITIALIZING], unassigned_info[[reason=CLUSTER_RECOVERED], at[2015-06-24T16:22:46.149Z]], indexUUID [RET0PNdWQQWhX_fglK4FIw], reason [shard failure [failed to update mappings][IllegalArgumentException[Mapper for [foo] conflicts with existing mapping in other types[mapper [foo] cannot be changed from type [string] to [date]]]]]
[2015-06-24 18:22:46,705][WARN ][indices.cluster          ] [Crossbones] [[bad][1]] marking and sending shard failed due to [failed to update mappings]
java.lang.IllegalArgumentException: Mapper for [foo] conflicts with existing mapping in other types[mapper [foo] cannot be changed from type [string] to [date]]
        at org.elasticsearch.index.mapper.FieldTypeLookup.checkCompatibility(FieldTypeLookup.java:117)
        at org.elasticsearch.index.mapper.MapperService.checkNewMappersCompatibility(MapperService.java:350)
        at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:305)
        at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:255)
        at org.elasticsearch.indices.cluster.IndicesClusterStateService.processMapping(IndicesClusterStateService.java:422)
        at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyMappings(IndicesClusterStateService.java:376)
        at org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:181)
        at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:484)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:209)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:179)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)
[2015-06-24 18:22:46,706][WARN ][cluster.action.shard     ] [Crossbones] [bad][0] received shard failed for [bad][0], node[_DucCUykRy6JHLfcchxVVg], [P], s[INITIALIZING], unassigned_info[[reason=CLUSTER_RECOVERED], at[2015-06-24T16:22:46.149Z]], indexUUID [RET0PNdWQQWhX_fglK4FIw], reason [master [Crossbones][_DucCUykRy6JHLfcchxVVg][Slim-2.local][inet[/127.0.0.1:9300]] marked shard as initializing, but shard is marked as failed, resend shard failure]
@szroland

This comment has been minimized.

Copy link
Contributor

commented Jun 28, 2015

A simpler way to reproduce this is to try put this mapping to ES master, and it will fail.
The ability to have a field with the same name in different types with uncompatible mapping seems to have been deliberately removed in #11812 by @rjernst.

@s1monw s1monw added the blocker label Jun 29, 2015

@clintongormley

This comment has been minimized.

Copy link
Member Author

commented Jun 29, 2015

@szroland the change is intentional. This issue is about what happens when you upgrade to 2.0 with conflicting mappings. We should fail once and stop, instead of trying over and over again.

@rjernst rjernst self-assigned this Jul 7, 2015

@imotov imotov assigned imotov and unassigned rjernst Jul 22, 2015

imotov added a commit to imotov/elasticsearch that referenced this issue Aug 5, 2015

Check for incompatible mappings while upgrading old indices
Conflicting mappings that were allowed before v2.0 can cause runaway shard failures on upgrade. This commit adds a check that prevents a cluster from starting if it contains such indices as well as restoring such indices from a snapshot into already running cluster.

Closes elastic#11857

@imotov imotov closed this in #12406 Aug 5, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.