Aggregations: toXContent response contains duplicate fields

[spinscale]

While debugging a watcher issue I encountered a problem that serializing a search response returned an exception saying that there were duplicate fields. Checking the JSON resulted in duplicate fields. I do not however understand why this is not an issue when writing out a search response, maybe the XContentHelper.convertToMap implementation is the culprit.

This happened on 6.3.2, but I also reproduced this up to master.

This is the test I was able to reproduce the issue with

    public void testFieldGetsWrittenOutTwice() throws Exception {
        // you need to add an additional index in order to trigger this (or potentially a shard)
        createIndex("foo_1");

        XContentBuilder builder = jsonBuilder().startObject().startObject("properties")
            .startObject("@timestamp").field("type", "date").endObject()
            .startObject("license").startObject("properties")
            .startObject("count").field("type", "long").endObject()
            .startObject("partnumber").field("type", "text").startObject("fields").startObject("keyword")
            .field("type", "keyword").field("ignore_above", 256)
            .endObject().endObject().endObject()
            .endObject().endObject().endObject().endObject();
        assertAcked(client().admin().indices().prepareCreate("foo_2")
            .addMapping("doc", builder).get());

        XContentBuilder docBuilder = jsonBuilder().startObject()
            .startObject("license").field("partnumber", "foobar").field("count", 2).endObject()
            .field("@timestamp", "2018-07-08T08:07:00.599Z")
            .endObject();

        client().prepareIndex("foo_2", "doc").setSource(docBuilder).setRefreshPolicy(WriteRequest.RefreshPolicy.IMMEDIATE).get();

        client().admin().indices().prepareRefresh();

        TermsAggregationBuilder groupByLicenseAgg = AggregationBuilders.terms("group_by_license_partnumber").field("license.partnumber.keyword");
        MaxBucketPipelineAggregationBuilder peakPipelineAggBuilder =
            PipelineAggregatorBuilders.maxBucket("peak", "licenses_per_day>total_licenses");
        SumAggregationBuilder sumAggBuilder = AggregationBuilders.sum("total_licenses").field("license.count");
        DateHistogramAggregationBuilder licensePerDayBuilder =
            AggregationBuilders.dateHistogram("licenses_per_day").field("@timestamp").dateHistogramInterval(DateHistogramInterval.DAY);
        licensePerDayBuilder.subAggregation(sumAggBuilder);
        groupByLicenseAgg.subAggregation(licensePerDayBuilder);
        groupByLicenseAgg.subAggregation(peakPipelineAggBuilder);

        SearchResponse response = client().prepareSearch("foo_*").setSize(0).addAggregation(groupByLicenseAgg).get();
        // you can see in the logging output that there are actually two peak fields
        // I do not yet understand why this is not a problem in our rest layer but only in this case
        logger.info(Strings.toString(response));
        // conversion to a map fails because there are two `peak` fields
        BytesReference bytes = XContentHelper.toXContent(response, XContentType.JSON, false);
        XContentHelper.convertToMap(bytes, false, XContentType.JSON);
    }

[elasticmachine]

Pinging @elastic/es-search-aggs