Support ordering by sub-aggregations for the terms-aggregator

[alexbrasetvik]

The terms-aggregator does not currently seem to fully support its documented orderings: it has troubles with ordering by sub-aggregators.

Here's an example that demonstrates the problem:

export ELASTICSEARCH_ENDPOINT=http://localhost:9200
ELASTICSEARCH_ENDPOINT=http://localhost:9200
curl -XPUT http://localhost:9200/foo -d '{
    "settings": {"index": {"number_of_shards": 5}},
    "mappings": {
        "type": {
            "properties": {
                "host": {
                    "type": "string",
                    "index": "not_analyzed"
                }
            }
        }
    }
}'
{"ok":true,"acknowledged":true}+ curl -XPOST 'http://localhost:9200/_bulk?refresh=true' -d '
{"index":{"_index":"foo","_type":"bar"}}
{"host":"app1","cpu":95.1,"timestamp":"2013-12-24T12:00:00Z"}
{"index":{"_index":"foo","_type":"bar"}}
{"host":"app1","cpu":75.1,"timestamp":"2013-12-24T12:01:00Z"}
{"index":{"_index":"foo","_type":"bar"}}
{"host":"app2","cpu":25.1,"timestamp":"2013-12-24T12:00:00Z"}
{"index":{"_index":"foo","_type":"bar"}}
{"host":"app2","cpu":55.1,"timestamp":"2013-12-24T12:01:00Z"}
'
{"took":364,"errors":false,"items":[{"create":{"_index":"foo","_type":"bar","_id":"dic5iqJRX2KimXqVq0tJg","_version":1,"ok":true,"status":201}},{"create":{"_index":"foo","_type":"bar","_id":"Gs1YrXtiSy-GE16bCSblng","_version":1,"ok":true,"status":201}},{"create":{"_index":"foo","_type":"bar","_id":"tfdbxfteSCO_ByUFiNYL6g","_version":1,"ok":true,"status":201}},{"create":{"_index":"foo","_type":"bar","_id":"kWRoKKqlSdOGWaZzEstziw","_version":1,"ok":true,"status":201}}]}
curl -XPOST 'http://localhost:9200/foo/_search?pretty' -d '
{
    "size": 0,
    "aggregations": {
        "host": {
            "terms": {
                "field": "host",
                "order": {
                    "cpu_avg": "asc"
                }
            },
            "aggs": {
                "cpu_avg": {
                    "avg": {
                        "field": "cpu"
                    }
                }
            }
        }
    }
}
'
{
  "took" : 164,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 4,
    "failed" : 1,
    "failures" : [ {
      "index" : "foo",
      "shard" : 0,
      "status" : 500,
      "reason" : "RemoteTransportException[[Piper][inet[/10.0.1.14:9301]][search/phase/query]]; nested: NullPointerException; "
    } ]
  },
  "hits" : {
    "total" : 1,
    "max_score" : 1.0,
    "hits" : [ ]
  },
  "aggregations" : {
    "host" : {
      "buckets" : [ {
        "key" : "app2",
        "doc_count" : 1,
        "cpu_avg" : {
          "value" : 25.1
        }
      } ]
    }
  }
}
+ curl -XPOST 'http://localhost:9200/foo/_search?pretty' -d '
{
    "size": 0,
    "aggregations": {
        "host": {
            "terms": {
                "field": "host",
                "order": {
                    "cpu.avg": "desc"
                }
            },
            "aggs": {
                "cpu": {
                    "stats": {
                        "field": "cpu"
                    }
                }
            }
        }
    }
}
'
{
  "took" : 12,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 4,
    "failed" : 1,
    "failures" : [ {
      "index" : "foo",
      "shard" : 0,
      "status" : 500,
      "reason" : "NullPointerException[null]"
    } ]
  },
  "hits" : {
    "total" : 1,
    "max_score" : 1.0,
    "hits" : [ ]
  },
  "aggregations" : {
    "host" : {
      "buckets" : [ {
        "key" : "app2",
        "doc_count" : 1,
        "cpu" : {
          "count" : 1,
          "min" : 25.1,
          "max" : 25.1,
          "avg" : 25.1,
          "sum" : 25.1
        }
      } ]
    }
  }
}

With the provided changes, I get the expected result:

export ELASTICSEARCH_ENDPOINT=http://localhost:9200
ELASTICSEARCH_ENDPOINT=http://localhost:9200
curl -XPUT http://localhost:9200/foo -d '{
    "settings": {"index": {"number_of_shards": 5}},
    "mappings": {
        "type": {
            "properties": {
                "host": {
                    "type": "string",
                    "index": "not_analyzed"
                }
            }
        }
    }
}'
{"ok":true,"acknowledged":true}+ curl -XPOST 'http://localhost:9200/_bulk?refresh=true' -d '
{"index":{"_index":"foo","_type":"bar"}}
{"host":"app1","cpu":95.1,"timestamp":"2013-12-24T12:00:00Z"}
{"index":{"_index":"foo","_type":"bar"}}
{"host":"app1","cpu":75.1,"timestamp":"2013-12-24T12:01:00Z"}
{"index":{"_index":"foo","_type":"bar"}}
{"host":"app2","cpu":25.1,"timestamp":"2013-12-24T12:00:00Z"}
{"index":{"_index":"foo","_type":"bar"}}
{"host":"app2","cpu":55.1,"timestamp":"2013-12-24T12:01:00Z"}
'
{"took":238,"errors":false,"items":[{"create":{"_index":"foo","_type":"bar","_id":"MGBw21a_RReC2_7Aos8g2g","_version":1,"ok":true,"status":201}},{"create":{"_index":"foo","_type":"bar","_id":"40JZSDw0SDevcm4ERm5ReA","_version":1,"ok":true,"status":201}},{"create":{"_index":"foo","_type":"bar","_id":"OZ4h_SQxRpeCWbxdta1dSA","_version":1,"ok":true,"status":201}},{"create":{"_index":"foo","_type":"bar","_id":"Bvrqvfs5T9ak-GEshuIqwQ","_version":1,"ok":true,"status":201}}]}+ curl -XPOST 'http://localhost:9200/foo/_search?pretty' -d '
{
    "size": 0,
    "aggregations": {
        "host": {
            "terms": {
                "field": "host",
                "order": {
                    "cpu_avg": "asc"
                }
            },
            "aggs": {
                "cpu_avg": {
                    "avg": {
                        "field": "cpu"
                    }
                }
            }
        }
    }
}
'
{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 4,
    "max_score" : 1.0,
    "hits" : [ ]
  },
  "aggregations" : {
    "host" : {
      "buckets" : [ {
        "key" : "app2",
        "doc_count" : 2,
        "cpu_avg" : {
          "value" : 40.1
        }
      }, {
        "key" : "app1",
        "doc_count" : 2,
        "cpu_avg" : {
          "value" : 85.1
        }
      } ]
    }
  }
}
curl -XPOST 'http://localhost:9200/foo/_search?pretty' -d '
{
    "size": 0,
    "aggregations": {
        "host": {
            "terms": {
                "field": "host",
                "order": {
                    "cpu.avg": "desc"
                }
            },
            "aggs": {
                "cpu": {
                    "stats": {
                        "field": "cpu"
                    }
                }
            }
        }
    }
}
'
{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 4,
    "max_score" : 1.0,
    "hits" : [ ]
  },
  "aggregations" : {
    "host" : {
      "buckets" : [ {
        "key" : "app1",
        "doc_count" : 2,
        "cpu" : {
          "count" : 2,
          "min" : 75.1,
          "max" : 95.1,
          "avg" : 85.1,
          "sum" : 170.2
        }
      }, {
        "key" : "app2",
        "doc_count" : 2,
        "cpu" : {
          "count" : 2,
          "min" : 25.1,
          "max" : 55.1,
          "avg" : 40.1,
          "sum" : 80.2
        }
      } ]
    }
  }
}

I'm not nearly familiar with the code to claim that these changes are sufficient, but it's a start at least. :)

[alexbrasetvik]

@jpountz @uboness A lot of PRs these days, so just mentioning you in this one, to prevent possible duplication of effort. No rush. :)

[uboness]

@alexbrasetvik ,

Do you mind if I take over this? I'd like to implement an optimization to avoid creating too many objects for nothing. I'll work on it, and incorporate all the changes you've made so far.

Cheers,
Uri

[alexbrasetvik]

Great! Go ahead :)

[alexbrasetvik]

Just took your latest fix for a spin. Seems to work well! Thanks a lot.

Closing this one, fixed by #4643