The significant_terms aggregation currently creates a cache of document counts per term. This is useful when there are multiple child-level significant_terms aggregations asking for the same background index frequency information for shared terms found in many different buckets.
However, for the simpler case when the significant_terms aggregation is used as a top-level aggregation examining the "single bucket" results of a query then the cache is not useful and so should not be constructed.

The logic for managing this cache is in SignificantTermsAggregatorFactory and a separate issue will be raised to address changing this cache into a new optimised utility class available for use elsewhere e.g. #5396