Endless re-syncing mappings

[heffergm]

We're running into an recurring issue of late where any change in cluster state (typically a node restart during an upgrade for example) results in the following errors streaming to the log of the currently elected master, as well as very high I/O on all eligible masters:

[2015-04-13 21:02:37,917][WARN ][cluster.metadata         ] [elasticsearch-master2.localdomain] [pelias] 
re-syncing mappings with cluster state for types [[osmway, locality, openaddresses, geonames
osmaddress, admin0, neighborhood, admin2, osmnode, admin1, local_admin]]

We are presently running 1.5.1, but we had the same issue on 1.5.0. I'm cannot say with certainty whether we experienced the issue under 1.4.x. The production cluster setup is as follows, but we have also seen this problem in local development (single node):

• 3 master eligible nodes
• 2 routing (no data, no master) nodes
• 8 data nodes
• 40 shards, no replicas, ~600gb of data

The problematic mapping seems to be the location part of the suggester field:

PUT pelias
{
  "mappings": {
    "admin0": {
      "properties": {
        "suggest": {
          "context": {
            "dataset": {
              "default": [],
              "path": "_type",
              "type": "category"
            },
            "location": {
              "default": [],
              "neighbors": true,
              "path": "center_point",
              "precision": [2,3,1,5,4],
              "type": "geo"
            }
          },
          "max_input_length": 50,
          "payloads": false,
          "preserve_position_increments": false,
          "preserve_separators": false,
          "type": "completion"
        }
      }
    }
  }
}

[clintongormley]

@areek please could you take a look at this. It appears to be the reordering of the precision field, as mentioned in #8937