different result on fasthll and distinctcounthll

[xuanzih]

hi guys, we are trying to switch from fasthll to distinctcounthll.
com.clearspring.analytics.stream.cardinality.HyperLogLog; is used in code and org.apache.pinot.core.startree.hll.HllUtil to serialize the hll to a string.
with the same condition we have 1000x difference.
Example:

SELECT fasthll(my_hll), distinctcounthll(my_hll)
FROM counts_table WHERE timestamp >= 1500768000

I get results:

"aggregationResults": [
    {
        "function": "fastHLL_my_hll",
        "value": "68685244"
    }, {
        "function": "distinctCountHLL_my_hll",
        "value": "50535"
    }]

Could anyone suggest what's the big difference between them?

[xiangfu0]

FastHll will convert one string into a hyperloglog object, which may represent thousand unique values. DistinctCountHLL treats string as a value, not hyperloglog object, so it will return the approximation of how many unique hyperloglog serialized strings, the value should be close to your total number scanned .

[xiangfu0]

DistinctCountHLL could only infer bytes column as a hyperloglog object.

[Jackie-Jiang]

fasthll is deprecated because of the low performance of deserialization. You may generate BYTES type for serialized HyperLogLog using org.apache.pinot.core.common.ObjectSerDeUtils.HYPER_LOG_LOG_SER_DE.serialize(hyperLogLog) and query it with distinctcounthll