After doing some profiling and seeing surprising results, and looking at the code, and I can easily be reading it wrong...

buildAggregation() has the following loop:

for (long globalTermOrd = Ordinals.MIN_ORDINAL; globalTermOrd < globalOrdinals.getMaxOrd(); ++globalTermOrd) {
                ...
                copy(globalValues.getValueByOrd(globalTermOrd), spare.termBytes);  
}

This is very costly for high-cardinality fields, because it means we lookup ord->term for every single one. Instead, can we use a PriorityQueue of "OrdAndSortValue", find the top-N, and then only at the end, lookup ord->term for the top-N? E.g. in solr faceting this OrdAndSortValue is just a long (32 bits ord and 32 bits count) but the representation is less important here.