Skip to content

High memory consumption of inverted index during merges/mutations #47875

@ardenwick

Description

@ardenwick

Describe the unexpected behaviour
As described in title, memory consumption increases insanely out of acceptable and causing servers to restart.

image

How to reproduce

  • Which ClickHouse server version to use
    ClickHouse Server 23.2.3.17 (Official Build)
  • Non-default settings, if any
SET allow_experimental_inverted_index=1

Expected behavior
memory consumption of inverted index be reasonabe.

Error message and/or stacktrace
this same error flushs my screen before server exits.

2023.03.22 15:30:55.022734 [ 60 ] {} <Error> business.asset_log_local::49e1a074a7d9fb28e20290014823a76e_0_1199_785 (MergeFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 241. DB::Exception: Memory limit (total) exceeded: would use 183.54 GiB (attempt to allocate chunk of 4994983 bytes), maximum: 172.80 GiB. OvercommitTracker decision: Memory overcommit isn't used. Waiting time or overcommit denominator are set to zero.: (while reading column data): While executing MergeTreeSequentialSource. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0xe0c67d5 in /usr/bin/clickhouse
1. ? @ 0xe0d20d1 in /usr/bin/clickhouse
2. MemoryTracker::allocImpl(long, bool, MemoryTracker*) @ 0xe0d1ab7 in /usr/bin/clickhouse
3. MemoryTracker::allocImpl(long, bool, MemoryTracker*) @ 0xe0d13d2 in /usr/bin/clickhouse
4. MemoryTracker::allocImpl(long, bool, MemoryTracker*) @ 0xe0d13d2 in /usr/bin/clickhouse
5. CurrentMemoryTracker::realloc(long, long) @ 0xe096e5f in /usr/bin/clickhouse
6. Allocator<false, false>::realloc(void*, unsigned long, unsigned long, unsigned long) @ 0x82c5187 in /usr/bin/clickhouse
7. ? @ 0xe127980 in /usr/bin/clickhouse
8. DB::CompressedReadBufferFromFile::nextImpl() @ 0x127e4f11 in /usr/bin/clickhouse
9. ? @ 0x1298de66 in /usr/bin/clickhouse
10. DB::ISerialization::deserializeBinaryBulkWithMultipleStreams(COW<DB::IColumn>::immutable_ptr<DB::IColumn>&, unsigned long, DB::ISerialization::DeserializeBinaryBulkSettings&, std::__1::shared_ptr<DB::ISerialization::DeserializeBinaryBulkState>&, std::__1::unordered_map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>*) const @ 0x1295484b in /usr/bin/clickhouse
11. DB::MergeTreeReaderCompact::readData(DB::NameAndTypePair const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn>&, unsigned long, unsigned long, unsigned long, unsigned long, std::__1::optional<unsigned long>) @ 0x143bf79f in /usr/bin/clickhouse
12. DB::MergeTreeReaderCompact::readRows(unsigned long, unsigned long, bool, unsigned long, std::__1::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::__1::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>&) @ 0x143bf0f3 in /usr/bin/clickhouse
13. DB::MergeTreeSequentialSource::generate() @ 0x143d1a51 in /usr/bin/clickhouse
14. DB::ISource::tryGenerate() @ 0x14747875 in /usr/bin/clickhouse
15. DB::ISource::work() @ 0x147473e6 in /usr/bin/clickhouse
16. DB::ExecutionThreadContext::executeTask() @ 0x14760d0c in /usr/bin/clickhouse
17. DB::PipelineExecutor::executeStepImpl(unsigned long, std::__1::atomic<bool>*) @ 0x14755f1b in /usr/bin/clickhouse
18. DB::PipelineExecutor::executeStep(std::__1::atomic<bool>*) @ 0x14755308 in /usr/bin/clickhouse
19. DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x14765dd7 in /usr/bin/clickhouse
20. DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x14765f93 in /usr/bin/clickhouse
21. DB::MergeTask::ExecuteAndFinalizeHorizontalPart::executeImpl() @ 0x1422b26f in /usr/bin/clickhouse
22. DB::MergeTask::ExecuteAndFinalizeHorizontalPart::execute() @ 0x1422b1cb in /usr/bin/clickhouse
23. DB::MergeTask::execute() @ 0x142303ba in /usr/bin/clickhouse
24. DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x144be86c in /usr/bin/clickhouse
25. DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x144bd6b4 in /usr/bin/clickhouse
26. DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::__1::shared_ptr<DB::TaskRuntimeData>) @ 0x8925a8c in /usr/bin/clickhouse
27. DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x892572a in /usr/bin/clickhouse
28. ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__1::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0xe198dca in /usr/bin/clickhouse
29. void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__1::__function::__policy_storage const*) @ 0xe19b697 in /usr/bin/clickhouse
30. ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0xe194f8a in /usr/bin/clickhouse
31. ? @ 0xe19a641 in /usr/bin/clickhouse
 (version 23.2.3.17 (official build))
2023.03.22 15:30:55.022838 [ 1 ] {} <Information> Application: Closed connections. But 157 remain. Tip: To increase wait time add to config: <shutdown_wait_unfinished>60</shutdown_wait_unfinished>
2023.03.22 15:30:55.022903 [ 1 ] {} <Information> Application: Will shutdown forcefully.

Metadata

Metadata

Assignees

No one assigned

    Labels

    featurememoryWhen memory usage is higher than expected

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions