Memory problem with array after update from 18.10.3 to 18.12.13

[ujiseven]

After update from 18.10.3 to 18.12.13(18.14.13) i have problem with query.

CREATE TABLE IF NOT EXISTS test (
    id String,
    myarray Array(String)
)
ENGINE = MergeTree()
    ORDER BY (sipHash64(id))
    SETTINGS index_granularity = 8192
; 

SELECT avg(length(myarray))
FROM test2

in 18.10.3 its work fine.
in 18.12.13(18.14.13) error:
Code: 241, e.displayText() = DB::Exception: Memory limit (for query) exceeded: would use 12.38 GiB (attempt to allocate chunk of 4294967296 bytes), maximum: 9.31 GiB: (while reading column myarray): (while reading from part /var/lib/clickhouse/data/default/test2/all_1_38_2/ from mark 84 with max_rows_to_read = 8192)

in log 18.12.13(18.14.13):

2018.11.15 14:58:40.004057 {a954b529-d17c-4b06-8293-50c794d1c0ce} [ 732 ] <Trace> Aggregator: Aggregation method: without_key
2018.11.15 14:58:40.004359 {a954b529-d17c-4b06-8293-50c794d1c0ce} [ 732 ] <Debug> MemoryTracker: Current memory usage: 4.45 GiB.
2018.11.15 14:58:40.011093 {a954b529-d17c-4b06-8293-50c794d1c0ce} [ 728 ] <Trace> Aggregator: Aggregation method: without_key
2018.11.15 14:58:40.011409 {a954b529-d17c-4b06-8293-50c794d1c0ce} [ 728 ] <Debug> MemoryTracker: Current memory usage: 8.41 GiB.
2018.11.15 14:58:40.011622 {a954b529-d17c-4b06-8293-50c794d1c0ce} [ 729 ] <Trace> Aggregator: Aggregation method: without_key

in log 18.10.3:

2018.11.15 20:15:18.455283 [ 64 ] <Trace> ParallelAggregatingBlockInputStream: Aggregating
2018.11.15 20:15:18.509659 [ 65 ] <Trace> Aggregator: Aggregation method: without_key
2018.11.15 20:15:18.539821 [ 66 ] <Trace> Aggregator: Aggregation method: without_key
2018.11.15 20:15:18.544867 [ 67 ] <Trace> Aggregator: Aggregation method: without_key
2018.11.15 20:15:18.545915 [ 68 ] <Trace> Aggregator: Aggregation method: without_key
2018.11.15 20:15:18.549937 [ 69 ] <Trace> Aggregator: Aggregation method: without_key
2018.11.15 20:15:18.560426 [ 70 ] <Trace> Aggregator: Aggregation method: without_key
2018.11.15 20:15:18.564289 [ 71 ] <Trace> Aggregator: Aggregation method: without_key
2018.11.15 20:15:18.565775 [ 72 ] <Trace> Aggregator: Aggregation method: without_key
2018.11.15 20:15:20.251781 [ 64 ] <Trace> ParallelAggregatingBlockInputStream: Aggregated. 229376 to 1 rows (from 1.750 MiB) in 1.796 sec. (127685.469 rows/sec., 0.974 MiB/sec.)
2018.11.15 20:15:20.251858 [ 64 ] <Trace> ParallelAggregatingBlockInputStream: Aggregated. 229376 to 1 rows (from 1.750 MiB) in 1.796 sec. (127685.469 rows/sec., 0.974 MiB/sec.)
2018.11.15 20:15:20.251870 [ 64 ] <Trace> ParallelAggregatingBlockInputStream: Aggregated. 229376 to 1 rows (from 1.750 MiB) in 1.796 sec. (127685.469 rows/sec., 0.974 MiB/sec.)
2018.11.15 20:15:20.251881 [ 64 ] <Trace> ParallelAggregatingBlockInputStream: Aggregated. 229376 to 1 rows (from 1.750 MiB) in 1.796 sec. (127685.469 rows/sec., 0.974 MiB/sec.)
2018.11.15 20:15:20.251892 [ 64 ] <Trace> ParallelAggregatingBlockInputStream: Aggregated. 229376 to 1 rows (from 1.750 MiB) in 1.796 sec. (127685.469 rows/sec., 0.974 MiB/sec.)
2018.11.15 20:15:20.251904 [ 64 ] <Trace> ParallelAggregatingBlockInputStream: Aggregated. 229376 to 1 rows (from 1.750 MiB) in 1.796 sec. (127685.469 rows/sec., 0.974 MiB/sec.)
2018.11.15 20:15:20.251915 [ 64 ] <Trace> ParallelAggregatingBlockInputStream: Aggregated. 229376 to 1 rows (from 1.750 MiB) in 1.796 sec. (127685.469 rows/sec., 0.974 MiB/sec.)
2018.11.15 20:15:20.251925 [ 64 ] <Trace> ParallelAggregatingBlockInputStream: Aggregated. 212771 to 1 rows (from 1.623 MiB) in 1.796 sec. (118442.055 rows/sec., 0.904 MiB/sec.)
2018.11.15 20:15:20.251950 [ 64 ] <Trace> ParallelAggregatingBlockInputStream: Total aggregated. 1818403 rows (from 13.873 MiB) in 1.796 sec. (1012240.335 rows/sec., 7.723 MiB/sec.)

in 18.10.3 MemoryTracker: Peak memory usage (total): 536.01 MiB.

Some information about my data:

SELECT avg(length(myarray))
FROM test2 

┌─avg(length(myarray))─┐
│     276.38314004101403 │
└────────────────────────┘

SELECT max(length(myarray))
FROM test2 

┌─max(length(myarray))─┐
│                   1054 │
└────────────────────────┘

SELECT count(*)
FROM test2 

┌─count()─┐
│ 1818403 │
└─────────┘

All version of clickhouse deployed from https://packagecloud.io/Altinity/clickhouse

# uname -a
Linux mytesthost 4.4.24-1.el7.elrepo.x86_64 #1 SMP Fri Oct 7 10:41:16 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux
# cat /proc/version 
Linux version 4.4.24-1.el7.elrepo.x86_64 (mockbuild@Build64R7) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Fri Oct 7 10:41:16 EDT 2016

Add all logs
logs.zip

[ujiseven]

Have same problem on my laptop

uname -a
Linux 3.19.0-25-generic #26~14.04.1-Ubuntu SMP Fri Jul 24 21:16:20 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

2018.11.15 21:30:33.254839 [ 99 ] {2642d9ca-49f1-4d0a-81f8-81e624d7a3e5} <Trace> ParallelAggregatingBlockInputStream: Aggregating
2018.11.15 21:30:33.288168 [ 101 ] {2642d9ca-49f1-4d0a-81f8-81e624d7a3e5} <Trace> Aggregator: Aggregation method: without_key
2018.11.15 21:30:33.288641 [ 101 ] {2642d9ca-49f1-4d0a-81f8-81e624d7a3e5} <Debug> MemoryTracker: Current memory usage: 4.07 GiB.
2018.11.15 21:30:33.304722 [ 100 ] {2642d9ca-49f1-4d0a-81f8-81e624d7a3e5} <Trace> Aggregator: Aggregation method: without_key
2018.11.15 21:30:33.304896 [ 100 ] {2642d9ca-49f1-4d0a-81f8-81e624d7a3e5} <Debug> MemoryTracker: Current memory usage: 8.04 GiB.

but its work because max_threads=2:
MemoryTracker: Peak memory usage (total): 8.07 GiB.

And have same problem on 18.14.13 (last stable)

[ujiseven]

when i set max_block_size = 100 its work on 18.12.13(18.14.13) but slowly

MemoryTracker: Peak memory usage (for query): 537.18 MiB.

[pilosus]

I've run into the same problem with arrays after updating ClickHouse from 18.10.3 to the latest stable. Queries with arrays that used to be performing well now crash with Memory limit (for query) exceeded. The exception seems to be thrown early, at the very beginning of execution

[Sahaquielxo]

Confirmed

[alexey-milovidov]

The bug was introduced in f56d167, #2830

[alexey-milovidov]

Fixed in master.

[alexey-milovidov]

In stable release.