Add MemoryTracker for the background tasks

[novikd]

Changelog category (leave one):

• New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Add MemoryTracker for the background tasks (merges and mutation). Introduces merges_mutations_memory_usage_soft_limit and merges_mutations_memory_usage_to_ram_ratio settings that represent the soft memory limit for merges and mutations. If this limit is reached ClickHouse won't schedule new merge or mutation tasks. Also MergesMutationsMemoryTracking metric is introduced to allow observing current memory usage of background tasks.
Closes #45710.

Documentation entry for user-facing changes

• Documentation is written (mandatory for new features)

[filimonov]

Check also
#10154 - there is a test there.

[tavplubix]

I'm afraid that it will be too easy to get "too many parts" if the soft limit is not high enough. Imagine that some big merges are currently executing and consume a big amount of memory. It will be impossible to assign small merges. So it would be better to make the limit flexible and give priority to small merges as we do in getMaxSourcePartsSizeForMerge.

However, it's more complicated for memory usage because memory usage should be O(1) and should not depend on the source parts size. The concern here is that big marges may hold memory for too long.

But there's a chance that it will just work as is (it's hard to predict), so let's just keep it in mind.

[cangyin]

sorry for any disturbution.

there may be some misbehaviors for this feature that I observed and want to report to you.

1. updated value for background_memory_usage_soft_limit is not reflected in system.sever_settings after config hot reload
2. metric BackgroundMemoryTracking is increasing in the long term:

SELECT
    now(),
    metric,
    formatReadableSize(value),
    description
FROM system.metrics
WHERE metric = 'MergesMutationsMemoryTracking'

Query id: 70f4c034-70b4-4a46-bc0e-6cdaf433c0cc

┌───────────────now()─┬─metric────────────────────────┬─formatReadableSize(value)─┬─description──────────────────────────────────────────────────────────────────────────┐
│ 2023-03-27 00:11:08 │ MergesMutationsMemoryTracking │ 134.15 GiB                │ Total amount of memory (bytes) allocated by background tasks (merges and mutations). │
└─────────────────────┴───────────────────────────────┴───────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────┘

while total available amount of memory is about 32 GiB

[novikd]

@cangyin thank you for the feedback. Indeed, I initially forgot to update BackgroundMemoryTracking metric after a background task finished, but I can't reproduce the issue with system.sever_settings you mentioned.

[tavplubix]

@novikd, you forgot to check the failed tests before merging