Aggregation returns 0 items

[thomastvedt]

I'm trying to get aggregations to work with NEST, but result.Aggs.Terms("name").Items is empty.. The JSON beeing sent to ElasticSearch looks like this:

{
  "size": 0,
  "aggs": {
    "calls_per_customer": {
      "terms": {
        "field": "customer.id",
        "size": 10,
        "order": {
          "total_duration": "desc"
        }
      },
      "aggs": {
        "customer_name": {
          "terms": {
            "script": "_source.customer.name",
            "size": 1
          }
        },
        "total_duration": {
          "sum": {
            "field": "duration"
          }
        },
        "average_duration": {
          "avg": {
            "field": "duration"
          }
        }
      }
    }
  },
  "query": {
    "match_all": {}
  },
  "filter": {
    "and": {
      "filters": [
        {
          "term": {
            "answered": true
          }
        },
        {
          "exists": {
            "field": "customer.id"
          }
        }
      ]
    }
  }
}

and produces the following result when querying manually in head-plugin:

took: 211
timed_out: false
_shards: {
total: 5
successful: 5
failed: 0
}
hits: {
total: 6691
max_score: 0
hits: [ ]
}
aggregations: {
calls_per_customer: {
buckets: [
{
key: ecd15513-3020-44e8-a3bd-XXX
doc_count: 865
total_duration: {
value: 262867
}
customer_name: {
buckets: [
{
key: Customer name 1
doc_count: 865
}
]
}
average_duration: {
value: 303.89248554913297
}
}
{
key: b716031f-18ae-4843-a34d-XXX
doc_count: 675
total_duration: {
value: 171038
}
customer_name: {
buckets: [
{
key: Customer name 2
doc_count: 675
}
]
}
average_duration: {
value: 253.38962962962964

.......

Am I doing something wrong here?

var esResult = _esClient.Search<ElasticCall>(s =>
{
    var query = s.Query(qc => qc.MatchAll());

    var filters = new List<FilterContainer>();
    filters.Add(Filter<ElasticCall>.Term(call => call.Answered, true));
    filters.Add(Filter<ElasticCall>.Exists("customer.id"));

    query = query.Size(0);

    if (callDirection.HasValue)
    {
        filters.Add(Filter<ElasticCall>.Term(call => call.Direction, callDirection));
    }
    if (fromDate.HasValue)
    {
        filters.Add(Filter<ElasticCall>.Range(rs => rs.GreaterOrEquals(fromDate)));
    }
    if (toDate.HasValue)
    {
        filters.Add(Filter<ElasticCall>.Range(r => r.LowerOrEquals(toDate)));
    }

    query.Aggregations(agg => 
        agg.Terms("calls_per_customer", termsAggDescriptor => termsAggDescriptor.Field(call=>call.Customer.Id).Size(limitTop ?? 10).Aggregations(
            aggs2 => 
                aggs2.Terms("customer_name", s1 => s1.Script("_source.customer.name").Size(1))
                .Sum("total_duration", s2 => s2.Field("duration"))
                .Average("average_duration", s3 => s3.Field("duration")))));


    var filtersAsFunctionArray = filters.Select(f => (Func<FilterDescriptor<ElasticCall>, FilterContainer>)((z) => f)).ToArray();
    var outerAndFilter = Filter<ElasticCall>.And(filtersAsFunctionArray);
    query.Filter(outerAndFilter);

    //JSON is ok and returns expected result
    var json = Encoding.UTF8.GetString(_esClient.Serializer.Serialize(query));
    return query;
});

//NEST returns the first aggregation
var callsPerCust = esResult.Aggs.Terms("calls_per_customer");
//But nothing inside..
var callsCount = callsPerCust.Items.Count; //0!

[jfren484]

Here's how I'm using Aggregations. In the result I'm using the Aggregations property instead of Aggs, which I didn't even realize was there until now (but when I debug it looks like it just adds one more layer on top of the Aggregations property).

var queryLocal = new List<FilterContainer>(filters.ToArray()) {Filter<AuthForReporting>.Term(t => t.AuthInput.AuthType, authType.ToString().ToLower())};

var result = ElasticClient
    .Search<AuthForReporting>(s => s.Size(0)
                                    .Query(q => q.Filtered(fq => fq.Filter(f => f.And(queryLocal.ToArray()))))
                                    .Aggregations(a => a.DateHistogram("AuthsByDate",
                                                                        t => t.Field(f => f.AuthResult.AuthEventDate)
                                                                                .Interval(DateInterval.Day.ToString().ToLower())
                                                                                .TimeZone(TimeZoneInfo.Local.BaseUtcOffset.ToEPCISTimeZoneOffsetFormat()))));

var bucket = result.Aggregations.Values.OfType<Bucket>().SingleOrDefault();
if (bucket != null)
{
    var row = new AuthByDateResult
                {
                    AuthType = authType,
                    DateAndValue = bucket.Items
                                        .OfType<HistogramItem>()
                                        .Select(i => new AuthByDateResultPart {Date = i.Date, Count = i.DocCount})
                };

    retval.Add(row);
}

When I was trying to figure out how it should work I just used the debugger inspectors to figure out which property had the data I needed to get out, and the concrete types (in my case, Bucket and HistogramItem - for a Terms aggregation the item type is KeyItem).

[thomastvedt]

I think the .Aggs is an extra layer which does some of the casting for you. In my case both .Aggregations.Items.Count and Aggs.Items.Count() is zero. But the bucket itself is there, and the underlying query looks valid.

[Mpdreamz]

Hi @thomastvedt first of all thank you for submitting a very detailed issue! Makes our jobs so much easier.

I did my best to try and reproduce the issue in this integration test:

https://github.com/elasticsearch/elasticsearch-net/blob/develop/src/Tests/Nest.Tests.Integration/Reproduce/Reproduce986Tests.cs

But I'm not seeing the same issue as you. This might be because the POCO that I inferred from your code is not accurate or because you are using a mapping that interferes somehow.

Can you have a look at the integration test?

[thomastvedt]

Well, this is kind of embarrassing... I studied the integrationtests (great as documentation btw), compared my code with the tests without finding any difference. Finally I checked fiddler, and realized that my queries were going to the wrong index which had no data... So completely my own fault using an incorrect connectionstring/indexname :-(

Works great now, thanks for your help :)

[Mpdreamz]

Awesome, glad to hear it wasn't a huge bug! Thanks for reporting nevertheless 👍