Endless mapping re-sync problem #7215

AVVS · 2014-08-09T09:30:09Z

We are running ES 1.3.1, but we had spotted this same issue on the previous versions as well
It can be solved temporarily by closing/opening index, but it will get back to such a state later

Corresponding mapping and index settings: https://gist.github.com/AVVS/bef59f42760256e2b5e8

Basically it looks like this and can last forever:

[2014-08-09 02:24:10,836][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:11,069][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:11,303][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:11,613][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:11,854][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:12,272][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:12,514][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:12,755][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:12,997][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:13,382][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:13,632][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:13,874][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:14,108][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:14,350][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:14,601][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:14,852][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:15,100][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:15,418][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:15,761][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:16,036][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:16,292][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:16,552][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:16,802][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:17,044][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:17,287][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:17,518][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:17,753][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:18,096][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:18,346][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:18,597][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:18,847][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:19,096][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:19,415][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:19,666][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:19,917][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:20,151][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:20,585][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:20,859][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:21,076][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:21,335][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:21,585][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:21,818][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:22,059][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:22,294][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]

The text was updated successfully, but these errors were encountered:

clintongormley · 2014-08-09T10:00:08Z

Hi @AVVS

How many nodes are you running? Could you add the relevant logs from the other nodes?

Do you have the same version of ES on all nodes? And the same version of the JVM? Same version of the ICU plugin?

I presume you are actively indexing into this index? Does this message stop if you stop indexing?

AVVS · 2014-08-09T10:17:47Z

48 nodes: 41 data, 4 http balancer, 3 master nodes
ICU: 2.3.0, all nodes
openjdk 7_u55, all nodes

I'm indexing a lot, but this message has, sadly, nothing to do with it. If I stop it persists. One easy way to trigger it sooner is to add more replicas to the index.

Other nodes dont really have anything interesting: Master nodes have last messages from the 4 days ago about adding a node to the cluster except for the actual elected master. HTTP nodes dont have anything, data nodes have GC message from some 8 hours ago

AVVS · 2014-08-09T10:25:05Z

One more thing: if I look at _cat/pending_tasks I notice that its at ~ 16,5k lines, a few lines are always about refreshing mappings, and the rest is the same message about dangling indices, but I guess it never gets to these tasks because its completely occupied with mapping refreshes

32662080    1s HIGH   refresh-mapping [profiles-2014-08-01][[profiles_v1]] 
32662081 892ms HIGH   refresh-mapping [profiles-2014-08-01][[profiles_v1]] 
32662082 850ms HIGH   refresh-mapping [profiles-2014-08-01][[profiles_v1]] 
...
32662084 845ms HIGH   refresh-mapping [profiles-2014-08-01][[profiles_v1]] 
...
4365952  3.7d NORMAL allocation dangled indices [test_index] 
...

clintongormley · 2014-08-09T10:32:41Z

If you close the dangling indices, does it stop then? Probably not, but worth a try.

For some reason the code in question thinks that your mapping has changed. I'm wondering if there is something that you've specified which isn't properly handled by the equals() method.
https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/cluster/metadata/MetaDataMappingService.java#L257

AVVS · 2014-08-09T10:36:34Z

I cant really close those dangling indices, they are not yet available:

{
   "error": "RemoteTransportException[[ubuntu74][inet[/10.10.100.74:9300]][indices/close]]; nested: IndexMissingException[[test_index] missing]; ",
   "status": 404
}

For some reason the code in question thinks that your mapping has changed. I'm wondering if there is something that you've specified which isn't properly handled by the equals() method.

What data can I provide to help with testing this?

clintongormley · 2014-08-09T11:04:05Z

Not sure yet - I'll come back to you with more.

clintongormley · 2014-08-11T10:46:43Z

I've reduced this to the following, which reproduces the problem:

PUT /t
{
  "mappings": {
    "profiles_v1": {
      "properties": {
        "fullName": {
          "type": "string",
          "fields": {
            "one": {
              "type": "string"
            },
            "two": {
              "type": "string"
            },
            "three": {
              "type": "string"
            },
            "completion": {
              "type": "string"
            },
            "four": {
              "type": "string"
            },
            "ngrams_front_omit_norms": {
              "type": "string"
            },
            "ngrams_back": {
              "type": "string"
            },
            "ngrams_back_omit_norms": {
              "type": "string"
            },
            "ngrams_middle": {
              "type": "string"
            },
            "shingle": {
              "type": "string"
            },
            "ngrams_front": {
              "type": "string"
            },
            "fullName": {
              "type": "string"
            }
          }
        }
      }
    }
  }
}

When it tries to compare the mappings here https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/cluster/metadata/MetaDataMappingService.java#L270 the fields are in a different order.

mapper.mappingSource returns the fields in the same order as above, while builders.mapping(type).source() moves the ngrams_middle field to the end:

{
  "profiles_v1": {
    "properties": {
      "fullName": {
        "type": "string",
        "fields": {
          "fullName": {
            "type": "string"
          },
          "two": {
            "type": "string"
          },
          "completion": {
            "type": "string"
          },
          "four": {
            "type": "string"
          },
          "three": {
            "type": "string"
          },
          "ngrams_front_omit_norms": {
            "type": "string"
          },
          "ngrams_back": {
            "type": "string"
          },
          "ngrams_back_omit_norms": {
            "type": "string"
          },
          "shingle": {
            "type": "string"
          },
          "one": {
            "type": "string"
          },
          "ngrams_front": {
            "type": "string"
          },
          "ngrams_middle": {
            "type": "string"
          }
        }
      }
    }
  }
}

AVVS · 2014-08-11T11:09:31Z

Ok, any easy fix to temporarily allow it to sync properly? i.e., put same mapping with different field order

clintongormley · 2014-08-11T12:16:16Z

@AVVS that may work, or changing field names might work as well. We're working on a fix.

…der to ensure that the source is always the same. Closes elastic#7215

…der to ensure that the source is always the same. Closes #7215

martijnvg · 2014-08-11T17:27:48Z

@AVVS Thanks for reporting this issue, the re-syncing of the mapping was causing by multi-fields not being serialised consistently and this is fixed now.

AVVS · 2014-08-11T17:32:03Z

Thanks, when should I expect 1.3.2 to be released? :)

martijnvg · 2014-08-11T17:33:31Z

@AVVS I expect a 1.3.2 release in the coming days.

…der to ensure that the source is always the same. Closes #7215

heffergm · 2015-04-13T20:46:12Z

I'm not entirely clear on the status of this bug, but wanted to report that under 1.5 and now 1.5.1 (potentially under 1.4.x as well, but I can't confirm that) we've run into the endless re-syncing mappings issue.

I can provide any further details you'd like, let me know.

clintongormley · 2015-04-14T13:02:45Z

@heffergm please could you open a new bug, and upload your mappings?

clintongormley · 2015-04-14T13:04:29Z

@heffergm I see you already have :) #10581

…der to ensure that the source is always the same. Closes elastic#7215

clintongormley self-assigned this Aug 9, 2014

clintongormley added the bug label Aug 11, 2014

clintongormley assigned martijnvg and unassigned clintongormley Aug 11, 2014

martijnvg added a commit to martijnvg/elasticsearch that referenced this issue Aug 11, 2014

Mappings: Make sure that multi fields are serialized in alphabetic or…

870eb35

…der to ensure that the source is always the same. Closes elastic#7215

martijnvg mentioned this issue Aug 11, 2014

Make sure that multi fields are serialized in a consistent order. #7220

Closed

martijnvg closed this as completed in a40cb16 Aug 11, 2014

martijnvg added a commit that referenced this issue Aug 11, 2014

Mappings: Make sure that multi fields are serialized in alphabetic or…

a24d8e8

…der to ensure that the source is always the same. Closes #7215

martijnvg added a commit that referenced this issue Aug 11, 2014

Mappings: Make sure that multi fields are serialized in alphabetic or…

60a9b7d

…der to ensure that the source is always the same. Closes #7215

martijnvg added a commit that referenced this issue Aug 11, 2014

Mappings: Make sure that multi fields are serialized in alphabetic or…

218ca66

…der to ensure that the source is always the same. Closes #7215

martijnvg added a commit that referenced this issue Sep 8, 2014

Mappings: Make sure that multi fields are serialized in alphabetic or…

326efe0

…der to ensure that the source is always the same. Closes #7215

mute pushed a commit to mute/elasticsearch that referenced this issue Jul 29, 2015

Mappings: Make sure that multi fields are serialized in alphabetic or…

bbaad68

…der to ensure that the source is always the same. Closes elastic#7215

mute pushed a commit to mute/elasticsearch that referenced this issue Jul 29, 2015

Mappings: Make sure that multi fields are serialized in alphabetic or…

698496f

…der to ensure that the source is always the same. Closes elastic#7215

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Endless mapping re-sync problem #7215

Endless mapping re-sync problem #7215

AVVS commented Aug 9, 2014

clintongormley commented Aug 9, 2014

AVVS commented Aug 9, 2014

AVVS commented Aug 9, 2014

clintongormley commented Aug 9, 2014

AVVS commented Aug 9, 2014

clintongormley commented Aug 9, 2014

clintongormley commented Aug 11, 2014

AVVS commented Aug 11, 2014

clintongormley commented Aug 11, 2014

martijnvg commented Aug 11, 2014

AVVS commented Aug 11, 2014

martijnvg commented Aug 11, 2014

heffergm commented Apr 13, 2015

clintongormley commented Apr 14, 2015

clintongormley commented Apr 14, 2015

Endless mapping re-sync problem #7215

Endless mapping re-sync problem #7215

Comments

AVVS commented Aug 9, 2014

clintongormley commented Aug 9, 2014

AVVS commented Aug 9, 2014

AVVS commented Aug 9, 2014

clintongormley commented Aug 9, 2014

AVVS commented Aug 9, 2014

clintongormley commented Aug 9, 2014

clintongormley commented Aug 11, 2014

AVVS commented Aug 11, 2014

clintongormley commented Aug 11, 2014

martijnvg commented Aug 11, 2014

AVVS commented Aug 11, 2014

martijnvg commented Aug 11, 2014

heffergm commented Apr 13, 2015

clintongormley commented Apr 14, 2015

clintongormley commented Apr 14, 2015