DROP/MODIFY COLUMN for compact part memory usage for tables with thousands of columns.

[UnamedRus]

If you have really wide table with many thousands of columns and you want to drop / modify column.
For wide parts it working fine, but for compact clickhouse would try to allocate a lot of memory and going to be killed via OOM killer.

Does it reproduce on recent release?

Yes, 21.9

How to reproduce

clickhouse-client -mn --query="SELECT 'CREATE TABLE xxx_really_wide( ' || arrayStringConcat(groupArray('column_'|| toString(number) || ' Nullable(UInt32)'), ',') || ') ENGINE=MergeTree ORDER BY assumeNotNull(column_0)' FROM numbers(6000) FORMAT TSVRaw" | clickhouse-client -mn

clickhouse-client -mn --query="SELECT arrayStringConcat(replicate('1', range(6000)), ',') FROM numbers(300) FORMAT TSVRaw" | clickhouse-client -mn --query "INSERT INTO xxx_really_wide FORMAT CSV " --send_logs_level='trace'

SELECT *
FROM system.merges
FORMAT Vertical

Query id: d9751847-a689-49f0-aab3-f868d9c08e34

Row 1:
──────
database:                    default
table:                       xxx_really_wide
elapsed:                     84.8952157
progress:                    0
num_parts:                   1
source_part_names:           ['all_1_1_0']
result_part_name:            all_1_1_0_3
source_part_paths:           ['/var/lib/clickhouse/data/default/xxx_really_wide/all_1_1_0/']
result_part_path:            /var/lib/clickhouse/data/default/xxx_really_wide/all_1_1_0_3/
partition_id:                all
is_mutation:                 1
total_size_bytes_compressed: 456027
total_size_marks:            2
bytes_read_uncompressed:     0
rows_read:                   0
bytes_written_uncompressed:  0
rows_written:                0
columns_written:             0
memory_usage:                16836533272
thread_id:                   16171
merge_type:
merge_algorithm:

Row 2:
──────
database:                    default
table:                       xxx_really_wide
elapsed:                     84.8947852
progress:                    0
num_parts:                   1
source_part_names:           ['all_2_2_0']
result_part_name:            all_2_2_0_3
source_part_paths:           ['/var/lib/clickhouse/data/default/xxx_really_wide/all_2_2_0/']
result_part_path:            /var/lib/clickhouse/data/default/xxx_really_wide/all_2_2_0_3/
partition_id:                all
is_mutation:                 1
total_size_bytes_compressed: 468027
total_size_marks:            2
bytes_read_uncompressed:     0
rows_read:                   0
bytes_written_uncompressed:  0
rows_written:                0
columns_written:             0
memory_usage:                16706294896
thread_id:                   16174
merge_type:
merge_algorithm:

[alexey-milovidov]

How much memory does it use before OOM?

[UnamedRus]

Around ~30GB as i understand

[276308.624413] Out of memory: Kill process 16736 (clickhouse-serv) score 976 or sacrifice child
[276308.625091] Killed process 16736 (clickhouse-serv) total-vm:49429712kB, anon-rss:31671880kB, file-rss:196308kB, shmem-rss:0kB

[alexey-milovidov]

We see that it's not enough for 6000 columns.
But how many columns this server can process successfully? 3000? 1500?

[UnamedRus]

But it also make fix for #6943 problem not addressing it fully, because if clickhouse would try to merge such compact parts you again need much more memory for that and problem just moved one step further from INSERT to merge and ALTERs.

So if you would have some concurrent merges happening, it could OOM even big servers.

But how many columns this server can process successfully? 3000? 1500?

For 3000 columns clickhouse would take 23.21 GiB of memory.

[alexey-milovidov]

I'm trying to improve it in #26929.

By the way, the issue should not exist if vertical merge algorithm is selected.
What algorithm is selected in your case?

[alexey-milovidov]

Looks like "vertical" algorithm cannot be used for compact parts.

[alexey-milovidov]

@CurtizJ It looks possible to enable vertical merge when doing Compact -> Wide parts.

[filimonov]

Any updates?

BTW: why OOM, not the memory exception? Why does it happen with compact parts (or it happens when they get converted to wide?)

[UnamedRus]

(or it happens when they get converted to wide?)

In order to drop column, ClickHouse tries to convert compact part to wide afaik.

[filimonov]

In order to drop column, ClickHouse tries to convert compact part to wide afaik.

But that means the same issue will happen when a few simultaneous (horizontal) merges will be triggered.

[alexey-milovidov]

No updates.