Mappings lost after cluster rolling restart

[rtoma]

Hi,

We have 3-node ELK cluster running 1.0.1. To activate a heapchange we did a node-by-node restart.

Unwanted result is loss of mappings on a small number of indices, causing errors like below when querying with Kibana:

Caused by: org.elasticsearch.search.SearchParseException: [logstash-pro-endeca-2014.04.13][3]: query[filtered((@source_host:"pro end app 006"))->cache(@timestamp:[2014-04-10T14:07:21+02:00 TO 2014-04-24T14:07:21+02:00])],from[0],size[50]: Parse Failure [No mapping fo
und for [@timestamp] in order to sort on]
        at org.elasticsearch.search.sort.SortParseElement.addSortField(SortParseElement.java:198)
        at org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:172)
        at org.elasticsearch.search.sort.SortParseElement.parse(SortParseElement.java:90)
        at org.elasticsearch.search.SearchService.parseSource(SearchService.java:583)
        ... 10 more

After copying the mapping from a sibling index everything is OK again.

[brwe]

Could be that you ran into this issue: #5623

Do you still have the mappings of logstash-pro-endeca-2014.04.13 from after the restart before copying the mappings back?

[rtoma]

That issue could be the cause for this issue, as we use the bulk api - in async replication mode - for continuous feeding of log events.

The mappings for this endeca index was "{}".
Another index with missing mappings, had this mapping:

{
  "logstash-pro-log4json-2014.04.14" : {
    "mappings" : {
      "log4json" : {
        "properties" : { }
      }
    }
  }
}

[brwe]

The missing mapping {} is the symptom of #5623 but the empty type ("log4json") in the other index is not. Also, I did not test with async mode.
I'll see if I can reproduce that.

[kimchy]

both can potentially be caused by #5623, thats not to say we might not have another bug, but I can see how both problems can happen because of it.

[brwe]

I have to admit that I do not see it yet. Can you give me a hint?

[rtoma]

I scanned the changelogs and found this bug fixed on 1.1.1 (April 16th). I guess its time to upgrade.