FiltersAggregation support breaks extraction of other aggregations

[amironoff]

FiltersAggregation support, introduced in #1121, breaks extraction of other aggregations from ES response. This affects the latest version in development (Tested on 1.4.0-ci20150102152352). Steps to reproduce:

1. Define a query with multiple aggregations: a. TermsAggregation, b. DateRangeAggregation (with multiple ranges) c. FiltersAggregation (with multiple filters defined and no sub-aggregations);
2. Execute a query and verify that aggregations were serialized as expected (e.g. using Fiddler);
3. Verify that the response contains computed aggregations (e.g. using Fiddler);
4. Observe that TermsAggregation was not extracted by Nest.

[radiosterne]

Gotta take a look at this, thanks for the tip!

[gmarz]

@amironoff any chance you could provide an actual example? I haven't been able to reproduce this myself. Here's what I've tried:

var response = this.Client.Search<ElasticsearchProject>(s => s
    .Aggregations(a => a
      .Terms("my-terms", t => t
        .Field(p => p.Name)
      )
      .DateRange("my-date-range", dr => dr
        .Field(p => p.StartedOn)
        .Ranges(
          rs => rs.From("19000101").To("now"),
          rs => rs.From("19500101").To("now")
        )
      )
      .Filters("my-filters", fs => fs
        .Filters(
          ffs => ffs.Term(p => p.Country, "Malaysia"),
          ffs => ffs.Term(p => p.Name, "elasticsearch")
        )
      )
    )
  );

  response.IsValid.Should().BeTrue();

  var myTerms = response.Aggs.Terms("my-terms");
  myTerms.Should().NotBeNull();
  myTerms.Items.Count.Should().BeGreaterThan(0);


  var myDateRange = response.Aggs.DateRange("my-date-range");
  myDateRange.Should().NotBeNull();
  myDateRange.Items.Count.Should().BeGreaterThan(0);

  var myFilters = response.Aggs.Filters("my-filters");
  myFilters.Should().NotBeNull();
  myFilters.Items.Count().Should().Be(2);

All passes.

@radiosterne have you looked into this at all yet and maybe have had some luck reproducing it?

[amironoff]

@gmarz Okay, I'll try to compile a full example. The ES version I'm using is 1.4.0. Is "this.Client" in your example a mock or an actual ES client?

[gmarz]

@amironoff great, thanks. It's an actual ES client.

[gmarz]

Hey @amironoff any luck in producing an example yet?

[radiosterne]

@gmarz every example I've tried works good, so I haven't been able to reproduce; waiting for @amironoff

[gmarz]

Thanks @radiosterne ! The aggs deserialization code being as brittle as it is, I would't be surprised at all if there is some edge case that @amironoff ran into that breaks it.

[radiosterne]

I absolutely agree, @gmarz, with our current implementation of stateless deserialization I think, actually, there are plenty of such cases :)

[amironoff]

Hi guys! The example is coming! Interestingly, ability to reproduce depends on the aggregation key and order of registering the aggs. More to come soon! :)

[amironoff]

Okay, please find a full example in https://github.com/amironoff/es-aggregation-bug/tree/master

[gmarz]

Wow @amironoff, this is great! I was expecting a simple test case, but you went the extra mile. This will make things much easier, thank you.

Looking into this now, stay tuned.

[gmarz]

@amironoff @radiosterne This ended up being another case of the aggregation response not being read to completion due to ordering of the results (similar to #1169 and #931). This only occurred when the filters portion of the response was returned first, which is why we had a hard time reproducing it.

Pushed the above fix and it'll be available in the next release. Thanks again @amironoff for the awesome repro. 👍

[amironoff]

@gmarz Great! I knew this was one of those bugs where an actual example will save everyone some time. Thanks for fixing that so quickly, @gmarz!