Freq Terms Enum

[kimchy]

A frequency caching terms enum, that also allows to be configured with an optional filter. To be used by both significant terms and phrase suggester.
This change extracts the frequency caching into the same code, and allow in the future to add a filter to control/customize the background frequencies

[kimchy]

Aye, it is still missing (should be added in another change) the ability to parse and provide the filter form the sig terms agg

[jpountz]

Should it be called in front of the for-loop instead?

[kimchy]

I wanted to only call it when we create the spare, otherwise we don't need it. Same logic applies to creating the spare lazily, no?

[jpountz]

Maybe the code is a bit confusing, but because insertion into the priority queue is done with insertWithOverflow, spare == null is going to be true on the first ordered.size() iterations of the loop and false afterwards. So here termsAggFactory.buildTermsEnum(context); would not be called only once but ordered.size() times.

[markharwood]

I wonder if the FreqTermsEnum needs splitting into two classes - one that handles filtering TTF and DF stats and a separate one that handles the responsibility of caching TTF and DF stats?
The SignificantTermsAggFactory has different scenarios where the decisions about filtered vs unfiltered and caching vs non caching are independent choices. If the significant terms agg is analysing a single bucket then caching is not required around calls to the underlying TermsEnum. If there is no "background stats context" defined for the significant terms (we need a new PR to design exactly how that will work) then there is no need for a filtered TermsEnum. These are independent choices.
So the ability to wrap a raw TermsEnum with layers that provide these features as required seems more appropriate.

[s1monw]

++ @markharwood