Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not use squashing for materialized views (fixes excessive memory usage) #34908

Closed
wants to merge 1 commit into from

Conversation

azat
Copy link
Collaborator

@azat azat commented Feb 25, 2022

Changelog category (leave one):

  • Bug Fix (user-visible misbehaviour in official stable or prestable release)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Do not use squashing for materialized views (fixes excessive memory usage)

During pushing data to views there is one interpreter for INSERT that is
created with squashing, that will not release memory for aggregate
states.

You can see this live leak by using jemalloc profiler and increasing
mmap threshold to 128MB (MMAP_THRESHOLD):

$ MALLOC_CONF=prof:true,prof_accum:true,lg_prof_interval:29 clickhouse-server &

$ jeprof --text clickhouse-server --base jeprof.*.0.* $(ls -t jeprof.*.heap | head -1) --focus DB::AggregateFunctionUniq::merge
Using local file clickhouse-server.
Using local file jeprof.130773.91.i91.heap.
addr2line: DWARF error: section .debug_info is larger than its filesize! (0x9b4c5 vs 0x865d0)
addr2line: DWARF error: section .debug_info is larger than its filesize! (0x5b43cc vs 0x429a58)
Total: 10270.3 MB
 10175.5 100.0% 100.0%  10175.5 100.0% Allocator::realloc
     0.0   0.0% 100.0%  10175.5 100.0% AllocatorWithStackMemory::realloc
     0.0   0.0% 100.0%  10175.5 100.0% DB::AggregateFunctionUniq::merge
     0.0   0.0% 100.0%  10175.5 100.0% DB::ColumnAggregateFunction::insertFrom
     0.0   0.0% 100.0%  10175.5 100.0% DB::ColumnAggregateFunction::insertMergeFrom@25cba0
     0.0   0.0% 100.0%  10175.5 100.0% DB::ColumnAggregateFunction::insertMergeFrom@25cc40
     0.0   0.0% 100.0%  10175.5 100.0% DB::ColumnAggregateFunction::insertRangeFrom
     0.0   0.0% 100.0%  10175.5 100.0% DB::ExceptionKeepingTransform::work
     0.0   0.0% 100.0%  10175.5 100.0% DB::ExecutionThreadContext::executeTask
     0.0   0.0% 100.0%  10175.5 100.0% DB::PipelineExecutor::execute
     0.0   0.0% 100.0%  10175.5 100.0% DB::PipelineExecutor::executeImpl
     0.0   0.0% 100.0%  10175.5 100.0% DB::PipelineExecutor::executeSingleThread
     0.0   0.0% 100.0%  10175.5 100.0% DB::PipelineExecutor::executeStepImpl
     0.0   0.0% 100.0%  10175.5 100.0% DB::SquashingChunksTransform::onConsume
     0.0   0.0% 100.0%  10175.5 100.0% DB::SquashingChunksTransform::work
     0.0   0.0% 100.0%  10175.5 100.0% DB::SquashingTransform::add
     0.0   0.0% 100.0%  10175.5 100.0% DB::SquashingTransform::addImpl
     0.0   0.0% 100.0%  10175.5 100.0% DB::SquashingTransform::append

Example of a query from tests:

:) insert into in_02223 select * from numbers(1e12) settings max_memory_usage='100Gi'

Before the patch:

2022.02.25 14:42:25.425536 [ 130786 ] {ac814e81-1430-4530-8aab-6ce1b2ad03fd} <Debug> MemoryTracker: Peak memory usage (for query): 20.46 GiB.
Ok.
Query was cancelled.
0 rows in set. Elapsed: 883.833 sec. Processed 1.33 billion rows, 10.63 GB (1.50 million rows/s., 12.02 MB/s.)

After:

2022.02.25 14:54:19.877537 [ 134271 ] {78461af3-ae1e-4a7f-a78d-a32b64152874} <Debug> MemoryTracker: Peak memory usage (for query): 94.43 MiB.
Ok.
Query was cancelled.
0 rows in set. Elapsed: 548.303 sec. Processed 886.02 million rows, 7.09 GB (1.62 million rows/s., 12.93 MB/s.)

Refs: #3796 (19.11.12.69+)

…sage)

During pushing data to views there is one interpreter for INSERT that is
created with squashing, that will not release memory for aggregate
states.

You can see this live leak by using jemalloc profiler and increasing
mmap threshold to 128MB (MMAP_THRESHOLD):

<details>

    $ MALLOC_CONF=prof:true,prof_accum:true,lg_prof_interval:29 clickhouse-server &

    $ jeprof --text clickhouse-server --base jeprof.*.0.* $(ls -t jeprof.*.heap | head -1) --focus DB::AggregateFunctionUniq::merge
    Using local file clickhouse-server.
    Using local file jeprof.130773.91.i91.heap.
    addr2line: DWARF error: section .debug_info is larger than its filesize! (0x9b4c5 vs 0x865d0)
    addr2line: DWARF error: section .debug_info is larger than its filesize! (0x5b43cc vs 0x429a58)
    Total: 10270.3 MB
     10175.5 100.0% 100.0%  10175.5 100.0% Allocator::realloc
         0.0   0.0% 100.0%  10175.5 100.0% AllocatorWithStackMemory::realloc
         0.0   0.0% 100.0%  10175.5 100.0% DB::AggregateFunctionUniq::merge
         0.0   0.0% 100.0%  10175.5 100.0% DB::ColumnAggregateFunction::insertFrom
         0.0   0.0% 100.0%  10175.5 100.0% DB::ColumnAggregateFunction::insertMergeFrom@25cba0
         0.0   0.0% 100.0%  10175.5 100.0% DB::ColumnAggregateFunction::insertMergeFrom@25cc40
         0.0   0.0% 100.0%  10175.5 100.0% DB::ColumnAggregateFunction::insertRangeFrom
         0.0   0.0% 100.0%  10175.5 100.0% DB::ExceptionKeepingTransform::work
         0.0   0.0% 100.0%  10175.5 100.0% DB::ExecutionThreadContext::executeTask
         0.0   0.0% 100.0%  10175.5 100.0% DB::PipelineExecutor::execute
         0.0   0.0% 100.0%  10175.5 100.0% DB::PipelineExecutor::executeImpl
         0.0   0.0% 100.0%  10175.5 100.0% DB::PipelineExecutor::executeSingleThread
         0.0   0.0% 100.0%  10175.5 100.0% DB::PipelineExecutor::executeStepImpl
         0.0   0.0% 100.0%  10175.5 100.0% DB::SquashingChunksTransform::onConsume
         0.0   0.0% 100.0%  10175.5 100.0% DB::SquashingChunksTransform::work
         0.0   0.0% 100.0%  10175.5 100.0% DB::SquashingTransform::add
         0.0   0.0% 100.0%  10175.5 100.0% DB::SquashingTransform::addImpl
         0.0   0.0% 100.0%  10175.5 100.0% DB::SquashingTransform::append

</details>

Example of a query from tests:

    :) insert into in_02223 select * from numbers(1e12) settings max_memory_usage='100Gi'

Before the patch:

    2022.02.25 14:42:25.425536 [ 130786 ] {ac814e81-1430-4530-8aab-6ce1b2ad03fd} <Debug> MemoryTracker: Peak memory usage (for query): 20.46 GiB.
    Ok.
    Query was cancelled.
    0 rows in set. Elapsed: 883.833 sec. Processed 1.33 billion rows, 10.63 GB (1.50 million rows/s., 12.02 MB/s.)

After:

    2022.02.25 14:54:19.877537 [ 134271 ] {78461af3-ae1e-4a7f-a78d-a32b64152874} <Debug> MemoryTracker: Peak memory usage (for query): 94.43 MiB.
    Ok.
    Query was cancelled.
    0 rows in set. Elapsed: 548.303 sec. Processed 886.02 million rows, 7.09 GB (1.62 million rows/s., 12.93 MB/s.)

Refs: ClickHouse#3796
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
@robot-clickhouse robot-clickhouse added the pr-bugfix Pull request with bugfix, not backported by default label Feb 25, 2022
@azat azat marked this pull request as draft February 25, 2022 16:18
@azat azat closed this Mar 5, 2022
@azat
Copy link
Collaborator Author

azat commented Mar 5, 2022

There is min_insert_block_size_rows_for_materialized_views/min_insert_block_size_bytes_for_materialized_views

@azat
Copy link
Collaborator Author

azat commented Mar 5, 2022

Initial problem that I faced up with should be fixed by #35072

@azat azat deleted the mv-leak branch November 19, 2022 11:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pr-bugfix Pull request with bugfix, not backported by default
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants