Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Endless mapping re-sync problem #7215

Closed
AVVS opened this Issue Aug 9, 2014 · 15 comments

Comments

Projects
None yet
4 participants
@AVVS
Copy link

commented Aug 9, 2014

  1. We are running ES 1.3.1, but we had spotted this same issue on the previous versions as well
  2. It can be solved temporarily by closing/opening index, but it will get back to such a state later

Corresponding mapping and index settings: https://gist.github.com/AVVS/bef59f42760256e2b5e8

Basically it looks like this and can last forever:

[2014-08-09 02:24:10,836][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:11,069][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:11,303][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:11,613][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:11,854][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:12,272][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:12,514][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:12,755][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:12,997][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:13,382][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:13,632][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:13,874][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:14,108][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:14,350][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:14,601][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:14,852][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:15,100][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:15,418][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:15,761][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:16,036][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:16,292][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:16,552][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:16,802][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:17,044][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:17,287][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:17,518][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:17,753][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:18,096][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:18,346][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:18,597][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:18,847][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:19,096][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:19,415][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:19,666][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:19,917][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:20,151][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:20,585][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:20,859][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:21,076][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:21,335][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:21,585][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:21,818][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:22,059][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
[2014-08-09 02:24:22,294][WARN ][cluster.metadata         ] [ubuntu74] [profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]]
@clintongormley

This comment has been minimized.

Copy link
Member

commented Aug 9, 2014

Hi @AVVS

How many nodes are you running? Could you add the relevant logs from the other nodes?

Do you have the same version of ES on all nodes? And the same version of the JVM? Same version of the ICU plugin?

I presume you are actively indexing into this index? Does this message stop if you stop indexing?

@clintongormley clintongormley self-assigned this Aug 9, 2014

@AVVS

This comment has been minimized.

Copy link
Author

commented Aug 9, 2014

  1. 48 nodes: 41 data, 4 http balancer, 3 master nodes
  2. ICU: 2.3.0, all nodes
  3. openjdk 7_u55, all nodes

I'm indexing a lot, but this message has, sadly, nothing to do with it. If I stop it persists. One easy way to trigger it sooner is to add more replicas to the index.

Other nodes dont really have anything interesting: Master nodes have last messages from the 4 days ago about adding a node to the cluster except for the actual elected master. HTTP nodes dont have anything, data nodes have GC message from some 8 hours ago

@AVVS

This comment has been minimized.

Copy link
Author

commented Aug 9, 2014

One more thing: if I look at _cat/pending_tasks I notice that its at ~ 16,5k lines, a few lines are always about refreshing mappings, and the rest is the same message about dangling indices, but I guess it never gets to these tasks because its completely occupied with mapping refreshes

32662080    1s HIGH   refresh-mapping [profiles-2014-08-01][[profiles_v1]] 
32662081 892ms HIGH   refresh-mapping [profiles-2014-08-01][[profiles_v1]] 
32662082 850ms HIGH   refresh-mapping [profiles-2014-08-01][[profiles_v1]] 
...
32662084 845ms HIGH   refresh-mapping [profiles-2014-08-01][[profiles_v1]] 
...
4365952  3.7d NORMAL allocation dangled indices [test_index] 
...
@clintongormley

This comment has been minimized.

Copy link
Member

commented Aug 9, 2014

If you close the dangling indices, does it stop then? Probably not, but worth a try.

For some reason the code in question thinks that your mapping has changed. I'm wondering if there is something that you've specified which isn't properly handled by the equals() method.
https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/cluster/metadata/MetaDataMappingService.java#L257

@AVVS

This comment has been minimized.

Copy link
Author

commented Aug 9, 2014

I cant really close those dangling indices, they are not yet available:

{
   "error": "RemoteTransportException[[ubuntu74][inet[/10.10.100.74:9300]][indices/close]]; nested: IndexMissingException[[test_index] missing]; ",
   "status": 404
}

For some reason the code in question thinks that your mapping has changed. I'm wondering if there is something that you've specified which isn't properly handled by the equals() method.

What data can I provide to help with testing this?

@clintongormley

This comment has been minimized.

Copy link
Member

commented Aug 9, 2014

Not sure yet - I'll come back to you with more.

@clintongormley

This comment has been minimized.

Copy link
Member

commented Aug 11, 2014

I've reduced this to the following, which reproduces the problem:

PUT /t
{
  "mappings": {
    "profiles_v1": {
      "properties": {
        "fullName": {
          "type": "string",
          "fields": {
            "one": {
              "type": "string"
            },
            "two": {
              "type": "string"
            },
            "three": {
              "type": "string"
            },
            "completion": {
              "type": "string"
            },
            "four": {
              "type": "string"
            },
            "ngrams_front_omit_norms": {
              "type": "string"
            },
            "ngrams_back": {
              "type": "string"
            },
            "ngrams_back_omit_norms": {
              "type": "string"
            },
            "ngrams_middle": {
              "type": "string"
            },
            "shingle": {
              "type": "string"
            },
            "ngrams_front": {
              "type": "string"
            },
            "fullName": {
              "type": "string"
            }
          }
        }
      }
    }
  }
}

When it tries to compare the mappings here https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/cluster/metadata/MetaDataMappingService.java#L270 the fields are in a different order.

mapper.mappingSource returns the fields in the same order as above, while builders.mapping(type).source() moves the ngrams_middle field to the end:

{
  "profiles_v1": {
    "properties": {
      "fullName": {
        "type": "string",
        "fields": {
          "fullName": {
            "type": "string"
          },
          "two": {
            "type": "string"
          },
          "completion": {
            "type": "string"
          },
          "four": {
            "type": "string"
          },
          "three": {
            "type": "string"
          },
          "ngrams_front_omit_norms": {
            "type": "string"
          },
          "ngrams_back": {
            "type": "string"
          },
          "ngrams_back_omit_norms": {
            "type": "string"
          },
          "shingle": {
            "type": "string"
          },
          "one": {
            "type": "string"
          },
          "ngrams_front": {
            "type": "string"
          },
          "ngrams_middle": {
            "type": "string"
          }
        }
      }
    }
  }
}
@AVVS

This comment has been minimized.

Copy link
Author

commented Aug 11, 2014

Ok, any easy fix to temporarily allow it to sync properly? i.e., put same mapping with different field order

@clintongormley

This comment has been minimized.

Copy link
Member

commented Aug 11, 2014

@AVVS that may work, or changing field names might work as well. We're working on a fix.

martijnvg added a commit to martijnvg/elasticsearch that referenced this issue Aug 11, 2014

Mappings: Make sure that multi fields are serialized in alphabetic or…
…der to ensure that the source is always the same.

Closes elastic#7215

@martijnvg martijnvg closed this in a40cb16 Aug 11, 2014

martijnvg added a commit that referenced this issue Aug 11, 2014

Mappings: Make sure that multi fields are serialized in alphabetic or…
…der to ensure that the source is always the same.

Closes #7215

martijnvg added a commit that referenced this issue Aug 11, 2014

Mappings: Make sure that multi fields are serialized in alphabetic or…
…der to ensure that the source is always the same.

Closes #7215

martijnvg added a commit that referenced this issue Aug 11, 2014

Mappings: Make sure that multi fields are serialized in alphabetic or…
…der to ensure that the source is always the same.

Closes #7215
@martijnvg

This comment has been minimized.

Copy link
Member

commented Aug 11, 2014

@AVVS Thanks for reporting this issue, the re-syncing of the mapping was causing by multi-fields not being serialised consistently and this is fixed now.

@AVVS

This comment has been minimized.

Copy link
Author

commented Aug 11, 2014

Thanks, when should I expect 1.3.2 to be released? :)

@martijnvg

This comment has been minimized.

Copy link
Member

commented Aug 11, 2014

@AVVS I expect a 1.3.2 release in the coming days.

martijnvg added a commit that referenced this issue Sep 8, 2014

Mappings: Make sure that multi fields are serialized in alphabetic or…
…der to ensure that the source is always the same.

Closes #7215
@heffergm

This comment has been minimized.

Copy link

commented Apr 13, 2015

I'm not entirely clear on the status of this bug, but wanted to report that under 1.5 and now 1.5.1 (potentially under 1.4.x as well, but I can't confirm that) we've run into the endless re-syncing mappings issue.

I can provide any further details you'd like, let me know.

@clintongormley

This comment has been minimized.

Copy link
Member

commented Apr 14, 2015

@heffergm please could you open a new bug, and upload your mappings?

@clintongormley

This comment has been minimized.

Copy link
Member

commented Apr 14, 2015

@heffergm I see you already have :) #10581

mute pushed a commit to mute/elasticsearch that referenced this issue Jul 29, 2015

Mappings: Make sure that multi fields are serialized in alphabetic or…
…der to ensure that the source is always the same.

Closes elastic#7215

mute pushed a commit to mute/elasticsearch that referenced this issue Jul 29, 2015

Mappings: Make sure that multi fields are serialized in alphabetic or…
…der to ensure that the source is always the same.

Closes elastic#7215
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.