Missing Aggregation in 1.4.0 throws ArrayOutOfBoundsException

[tylerprete]

This query used to work fine on 1.3.5:

{
  "from": 0,
  "size": 0,
  "query": {
    "filtered": {
      "query": {
        "match_all": {}
      },
      "filter": {
        "bool": {
          "must": {
            "term": {
              "closed": false
            }
          },
          "_cache": true
        }
      }
    }
  },
  "aggregations": {
    "missing-external_link-square": {
      "missing": {
        "field": "external_link.square"
      }
    }
  }
}

I upgraded the index to 1.4.0, and now it returns this stack trace (on 10 shards):

{
"error": "SearchPhaseExecutionException[Failed to execute phase [query], all shards failed; shardFailures {[LBJC8-OYQySmdIMINrn-Ow][radius_2014-11-13-23-39_updated][0]: QueryPhaseExecutionException[[radius_2014-11-13-23-39_updated][0]: query[filtered(ConstantScore(cache(BooleanFilter(+cache(closed:F)))))->cache(_type:place)],from[0],size[0]: Query Failed [Failed to execute main query]]; nested: ArrayIndexOutOfBoundsException; }{[LBJC8-OYQySmdIMINrn-Ow][radius_2014-11-13-23-39_updated][1]: QueryPhaseExecutionException[[radius_2014-11-13-23-39_updated][1]: query[filtered(ConstantScore(cache(BooleanFilter(+cache(closed:F)))))->cache(_type:place)],from[0],size[0]: Query Failed [Failed to execute main query]]; nested: ArrayIndexOutOfBoundsException; }{[LBJC8-OYQySmdIMINrn-Ow][radius_2014-11-13-23-39_updated][2]: QueryPhaseExecutionException[[radius_2014-11-13-23-39_updated][2]: query[filtered(ConstantScore(cache(BooleanFilter(+cache(closed:F)))))->cache(_type:place)],from[0],size[0]: Query Failed [Failed to execute main query]]; nested: ArrayIndexOutOfBoundsException; }{[LBJC8-OYQySmdIMINrn-Ow][radius_2014-11-13-23-39_updated][3]: QueryPhaseExecutionException[[radius_2014-11-13-23-39_updated][3]: query[filtered(ConstantScore(cache(BooleanFilter(+cache(closed:F)))))->cache(_type:place)],from[0],size[0]: Query Failed [Failed to execute main query]]; nested: ArrayIndexOutOfBoundsException; }{[LBJC8-OYQySmdIMINrn-Ow][radius_2014-11-13-23-39_updated][4]: QueryPhaseExecutionException[[radius_2014-11-13-23-39_updated][4]: query[filtered(ConstantScore(cache(BooleanFilter(+cache(closed:F)))))->cache(_type:place)],from[0],size[0]: Query Failed [Failed to execute main query]]; nested: ArrayIndexOutOfBoundsException; }{[LBJC8-OYQySmdIMINrn-Ow][radius_2014-11-13-23-39_updated][5]: QueryPhaseExecutionException[[radius_2014-11-13-23-39_updated][5]: query[filtered(ConstantScore(cache(BooleanFilter(+cache(closed:F)))))->cache(_type:place)],from[0],size[0]: Query Failed [Failed to execute main query]]; nested: ArrayIndexOutOfBoundsException; }{[LBJC8-OYQySmdIMINrn-Ow][radius_2014-11-13-23-39_updated][6]: QueryPhaseExecutionException[[radius_2014-11-13-23-39_updated][6]: query[filtered(ConstantScore(cache(BooleanFilter(+cache(closed:F)))))->cache(_type:place)],from[0],size[0]: Query Failed [Failed to execute main query]]; nested: ArrayIndexOutOfBoundsException; }{[LBJC8-OYQySmdIMINrn-Ow][radius_2014-11-13-23-39_updated][7]: QueryPhaseExecutionException[[radius_2014-11-13-23-39_updated][7]: query[filtered(ConstantScore(cache(BooleanFilter(+cache(closed:F)))))->cache(_type:place)],from[0],size[0]: Query Failed [Failed to execute main query]]; nested: ArrayIndexOutOfBoundsException; }{[LBJC8-OYQySmdIMINrn-Ow][radius_2014-11-13-23-39_updated][8]: QueryPhaseExecutionException[[radius_2014-11-13-23-39_updated][8]: query[filtered(ConstantScore(cache(BooleanFilter(+cache(closed:F)))))->cache(_type:place)],from[0],size[0]: Query Failed [Failed to execute main query]]; nested: ArrayIndexOutOfBoundsException; }{[LBJC8-OYQySmdIMINrn-Ow][radius_2014-11-13-23-39_updated][9]: QueryPhaseExecutionException[[radius_2014-11-13-23-39_updated][9]: query[filtered(ConstantScore(cache(BooleanFilter(+cache(closed:F)))))->cache(_type:place)],from[0],size[0]: Query Failed [Failed to execute main query]]; nested: ArrayIndexOutOfBoundsException; }]",
"status": 500
}

[s1monw]

@jpountz can you take a look at this?

[jpountz]

@tylerprete Can you share the mappings of your external_link.square field? If you look at the lines that have been written in the elasticsearch log file at the same time that you got this error, you should see a stack trace for the ArrayIndexOutOfBoundsException, this information would be very helpful too. Thanks!

[tylerprete]

Thanks for looking into this so quickly!

Mapping for that field is nothing fancy:

{
  "external_link": {
    "properties": {
      "square": {
        "type": "string"
      }
    }
  }
}

I believe this is the stack trace you're looking for:

[2014-11-24 04:41:17,894][DEBUG][action.search.type       ] [10.0.4.38] [radius_2014-11-13-23-39_updated][5], node[AlrEzws1TiiuZ6CiC-9RvQ], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@3e546cc6] lastShard [true]
org.elasticsearch.search.query.QueryPhaseExecutionException: [radius_2014-11-13-23-39_updated][5]: query[filtered(ConstantScore(cache(BooleanFilter(+cache(closed:F)))))->cache(_type:place)],from[0],size[0]: Query Failed [Failed to execute main query]
        at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:163)
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:275)
        at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
        at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
        at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 728
        at org.apache.lucene.util.packed.Direct16.get(Direct16.java:54)
        at org.apache.lucene.util.packed.PackedLongValues.get(PackedLongValues.java:102)
        at org.apache.lucene.util.packed.PackedLongValues.get(PackedLongValues.java:110)
        at org.elasticsearch.index.fielddata.ordinals.MultiOrdinals$MultiDocs.ordAt(MultiOrdinals.java:166)
        at org.elasticsearch.index.fielddata.AbstractRandomAccessOrds.nextOrd(AbstractRandomAccessOrds.java:41)
        at org.apache.lucene.index.DocValues$5.get(DocValues.java:171)
        at org.elasticsearch.search.aggregations.bucket.missing.MissingAggregator.collect(MissingAggregator.java:59)
        at org.elasticsearch.search.aggregations.BucketCollector$2.collect(BucketCollector.java:81)
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.collectBucketNoCounts(BucketsAggregator.java:74)
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.collectExistingBucket(BucketsAggregator.java:63)
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.collectBucket(BucketsAggregator.java:55)
        at org.elasticsearch.search.aggregations.bucket.filter.FilterAggregator.collect(FilterAggregator.java:61)
        at org.elasticsearch.search.aggregations.AggregationPhase$AggregationsCollector.collect(AggregationPhase.java:161)
        at org.elasticsearch.common.lucene.MultiCollector.collect(MultiCollector.java:60)
        at org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:193)
        at org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:163)
        at org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)
        at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)
        at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:117)
        ... 7 more