Ability to ingest pre-computed hyperUnique value

[DaimonPl]

We've got druid data ingestion use case in which pre-computed HLL indexes can be stored in approximately 6*10^4 rows but preparing same data in raw format and ingesting it in druid requires preparation of 10^12 rows.

Because of this we've switched to thetaSketch aggregator which allows loading of pre-computer theta sketch.

However in our use case we only need count distinct feature so theta sketch itself adds too much overhead (it's much more space consuming than hyperUnique).

It would be great if:

• druid would allow ingesting of pre-computed hyperUnique (like thetaSketch does)
• druid would expose public API which allows to compute and merge hyperUnique values so it can be precomputed in other systems (for example in Hive using UDF)

[gianm]

Wow, you have 16 million unique values per row? That's some good rollup :)

This seems like a useful feature, would you be willing to contribute code for it? I think what's needed is a new ComplexMetricsSerde for hyperUnique that treats strings as base64 encoded collectors, rather than as plain strings, and a flag on the aggregator factory that changes the return value of getTypeName to trigger that alternate serde. Sort of like how theta sketches work.

Then, breaking out the HLL algorithm into a druid-hll module with minimal dependencies would make it easy to include in other projects.

[DaimonPl]

Original dataset is not that big but it must be self-joined (on quite 'explosive' condition) with hyperUnique values on both sides. So if those hyperUnique values can be pre-computed before that join, then resulting dataset to ingest is small. Unfortunately because of way current hyperUnique ingestion is done we'd need to do that 'explosive' join (it's not as bad as cartesian, but still bad ;) ) first and put result to druid to ingest - and that result dataset would be simply too big.

I haven't dived into druid internal code yet (at all) so I'm not sure how long would it take to understand changes you are talking about but I can try :)

[DaimonPl]

I'll be working on this. Is there some way to contact you in case i have some doubts?

[gianm]

Sure, just post here!

[DaimonPl]

First PR available (extraction of HLL code to separate module)

[DaimonPl]

One more question.

As I understand those changes will be available in druid 0.10.0. currently we are using 0.9.2 - is there any way to get it included in older branch (other than our own custom fork with merge) ?

Is it posible to merge it to 0.9.3 ? If yes when do you think 0.9.3 would be released?

[DaimonPl]

@gianm could you help about this last question?

[gianm]

Yep your patch will be in 0.10.0. That's the next planned release, there won't be an 0.9.3 barring unforeseen circumstances.

[DaimonPl]

Ok, closing issue itself

[amitme10]

Hey ,
I hope it's related :

I'm trying to use this aggregator :
{
"type" : "hyperUnique",
"name" : <output_name>,
"fieldName" : <metric_name>,
"isInputHyperUnique" : true,
"round" : false
}

on my precomputed sketch, which was built with swoop's spark-alchemy hll functions.

However I'm struggling with an "illegal base64 character 3c" error there .

I wonder what's the problem ... maybe the format is not correct there or something with my loading data process is wrong?

[DaimonPl]

You need to use druid-hll for that and encode it with base 64

[amitme10]

Thanks @DaimonPl , I added the druid-hll dependency to my pom .

you mean I should use a specific function from there on my sketch , and then encode it to base64 (in the end of course add it to the DF)?

If so, could you kindly tell me which function?