Tantivy uses a hll sketch for cardinality. This sketch is supposed to adapt itself internally for low cardinality.
First it lists coupons, then it upgrade to a set, then it upgrade to an actual hll sketch.
We observe this behavior in tantivy, and counts seem correct after union.
In Quickwit however, the results are wrong as highlighted by this test.
https://github.com/quickwit-oss/quickwit/blob/main/quickwit/rest-api-tests/scenarii/aggregations/0001-aggregations.yaml#L339-L366
(assert should be passing with an equality test)
We do not know yet the cause. (serialization code promoting to HLL? union code not doing what it is supposed to do?)
Tantivy uses a hll sketch for cardinality. This sketch is supposed to adapt itself internally for low cardinality.
First it lists coupons, then it upgrade to a set, then it upgrade to an actual hll sketch.
We observe this behavior in tantivy, and counts seem correct after union.
In Quickwit however, the results are wrong as highlighted by this test.
https://github.com/quickwit-oss/quickwit/blob/main/quickwit/rest-api-tests/scenarii/aggregations/0001-aggregations.yaml#L339-L366
(assert should be passing with an equality test)
We do not know yet the cause. (serialization code promoting to HLL? union code not doing what it is supposed to do?)