CLICKHOUSE-3547 streaming histogram aggregation

[ssmike]

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

[KochetovNicolai]

Macro is used for single case. Why don't write bins_count = applyVisitor(FieldVisitorConvertToNumber<UInt32>(), params[0]);?

[KochetovNicolai]

It's better to add ErrorCode.
It's not obvious that bins count is a parameter (not an argument), and we must specify it. For example:

:) select histogram(number) from (select * from system.numbers limit 20);

SELECT histogram(number)
FROM 
(
    SELECT *
    FROM system.numbers 
    LIMIT 20
) 

Received exception from server (version 1.1.54386):
Code: 0. DB::Exception: Received from localhost:9004, ::1. DB::Exception: Function histogram requires only bins count.

[KochetovNicolai]

Why do we use int here? max_bins and size are UInt32.

[KochetovNicolai]

We will use foreign memory if max_bins == 0.

[ssmike]

max_bins=0 is strange case. I think function should accept only positive bin count.

[KochetovNicolai]

Also we need to check that argument is numeric.

[alexey-milovidov]

Simple test for performance and precision:

SELECT arrayJoin(histogram(10)(number)) AS res, res.3 / (res.2 - res.1) AS density, bar(density * 1000, 0, 1000, 80) AS bar FROM numbers(1000000000)

it's about 8.5 million rows/sec per single core.
The bottleneck is compress function and, consequently, priority_queue.

[alexey-milovidov]

SELECT 
    CounterID, 
    count(), 
    histogramIf(10)(SendTiming, SendTiming != -1)
FROM hits_1000m_single 
GROUP BY CounterID
ORDER BY count() DESC
LIMIT 20

Looks like the results are wrong:

232261 │ 12148031 │ [(0,2.2250738585072014e-308,12148031)]

[alexey-milovidov]

:) SELECT histogram(10)(-2.0)

SELECT histogram(10)(-2.)

┌─histogram(10)(-2.)───────────────┐
│ [(-2,2.2250738585072014e-308,1)] │
└──────────────────────────────────┘

[alexey-milovidov]

https://en.cppreference.com/w/cpp/types/numeric_limits/min

[alexey-milovidov]

The values don't look real on this query:

┌─CounterID─┬──count()─┬─quantilesTimingIf(0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1)(SendTiming, greater(SendTiming, 0))─┬─histogramIf(10)(SendTiming, greater(SendTiming, 0))──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│    114208 │ 56057344 │ [1,78,112,149,193,248,322,433,621,1072,30000]                                                            │ [(1,20215729.61420258,14308657.75),(20215729.61420258,52954898.79166667,2044103.125),(52954898.79166667,82019633.16666667,3.875),(82019633.16666667,111713231.25,1.375),(111713231.25,150951663.75,1.75),(150951663.75,239427425,1.125),(239427425,326503053,1),(326503053,472743336.5,1),(472743336.5,2076058731.5,1),(2076058731.5,2227775,1)]

[alexey-milovidov]

TODO: allocate all state inplace, get rid of WeightedValue * pointer.
Look for uniqUpTo aggregate function as an example.

[alexey-milovidov]

The type of parameter is not checked:

SELECT CounterID, count(), histogramIf(-1)(SendTiming, SendTiming > 0) FROM hits_1000m_single GROUP BY CounterID ORDER BY count() DESC LIMIT 20

Memory limit (for query) exceeded: would use 128.02 GiB

[alexey-milovidov]

As we preallocate memory for maximum size of histogram, better to limit its size by 256 (or maybe 1000).

[alexey-milovidov]

Incorrect error message:

:) SELECT histogram(0.5)(1)

SELECT histogram(0.5)(1)

Received exception from server (version 1.1.54388):
Code: 36. DB::Exception: Received from localhost:9000, ::1. DB::Exception: Bin count should be positive.

[alexey-milovidov]

This should throw an exception:
SELECT histogram(1.5)(1)

[ssmike]

SELECT topK(0.2)(number) FROM (SELECT * FROM system.numbers LIMIT 50)

Received exception from server (version 1.1.54387):
Code: 69. DB::Exception: Received from localhost:9000, ::1. DB::Exception: Parameter 0 is illegal for aggregate function topK.

¯\_(ツ)_/¯

[ssmike]

Ok, I've fixed limits. But I don't understand what you expect to see in histogram for one value.

[fizerkhan]

I am confused, how to use it for following scenerio

I have table with two columns such as route name and response time as duration. I want to see the histogram of response time with maximum duration of 5 second with 10 buckets.

is this possible?

histogram(5, 10)(duration)

[alexey-milovidov]

histogramIf(10)(duration, duration <= 5)

Here we applied -If combinator.
Read more: https://clickhouse.yandex/docs/en/query_language/agg_functions/combinators/

[fizerkhan]

@alexey-milovidov Awesome. That makes sense.

Let's say If don't use the -If combinator, then how the behaviour will be

histogram(10)(duration)

Here I can understand that 10 ranges will come, but what is the ceiling limit?

[alexey-milovidov]

Histogram will span between actual minimum and maximum of data value.

[fizerkhan]

@alexey-milovidov Great. Thanks

[nanohayder]

Hi,

is it possible to histogram return fixed size ranges within the number of buckets specified for example 0-5,5-10,10-15,... that would be very helpful

Thanks

[fizerkhan]

@alexey-milovidov We have the following table

count  UInt64
avg   Float32

We could like something like

select sum(count), histogramIf(10)(avg, avg <= 20) from summingtable

We would like to do get both the sum of count and the histogram together