Make system.replicas parallel

[evillique]

Changelog category (leave one):

• Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Make system.replicas table do parallel fetches of replicas statuses. Closes #43918

[tavplubix]

Also consider using MultiRead requests in StorageReplicatedMergeTree::getStatus. Currently it does 3 + number_of_replicas (usually 3) = 6 subsequent synchronous requests = 6 RTT. We can replace it with 1 MultiGetChildren and 1 MultiGet subsequent requests (2 RTT, 3 times faster).

[tavplubix]

See the comments above

[tavplubix]

LGTM now, but it would be great to optimize StorageReplicatedMergeTree::getStatus as well

[evillique]

Also consider using MultiRead requests in StorageReplicatedMergeTree::getStatus. Currently it does 3 + number_of_replicas (usually 3) = 6 subsequent synchronous requests = 6 RTT. We can replace it with 1 MultiGetChildren and 1 MultiGet subsequent requests (2 RTT, 3 times faster).

From the code of getStatus() it looks like it will be 1 MultiGetChildren, 1 standard Get, and 1 MultiExists. I will try to optimize it.

[tavplubix]

1 standard Get, and 1 MultiExists

Exists can be replaced with Get and error code check

[tavplubix]

It was not necessary to merge master and restart CI

[tavplubix]

Integration tests (asan) [2/6] - #44928 and #44908
Integration tests (asan) [4/6] - #45160
Integration tests (release) [4/4] - #44908
Integration tests (tsan) [2/6] - #44908
Stress test (asan) - #44910

Stress test (ubsan) - #45251, also there are tons of errors like

2023.01.05 07:43:05.844601 [ 1677 ] {} <Error> void DB::MergeTreeBackgroundExecutor<DB::MergeMutateRuntimeQueue>::routine(DB::TaskRuntimeDataPtr) [Queue = DB::MergeMutateRuntimeQueue]: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 34.33 GiB (attempt to allocate chunk of 1146964 bytes), maximum: 34.29 GiB. OvercommitTracker decision: Memory overcommit isn't used. Waiting time or overcommit denominator are set to zero. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):
2023.01.05 07:43:05.855458 [ 1686 ] {} <Error> void DB::MergeTreeBackgroundExecutor<DB::MergeMutateRuntimeQueue>::routine(DB::TaskRuntimeDataPtr) [Queue = DB::MergeMutateRuntimeQueue]: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 34.33 GiB (attempt to allocate chunk of 1051048 bytes), maximum: 34.29 GiB. OvercommitTracker decision: Memory overcommit isn't used. Waiting time or overcommit denominator are set to zero. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):
2023.01.05 07:43:05.860716 [ 1681 ] {} <Error> void DB::MergeTreeBackgroundExecutor<DB::MergeMutateRuntimeQueue>::routine(DB::TaskRuntimeDataPtr) [Queue = DB::MergeMutateRuntimeQueue]: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 34.33 GiB (attempt to allocate chunk of 1072930 bytes), maximum: 34.29 GiB. OvercommitTracker decision: Memory overcommit isn't used. Waiting time or overcommit denominator are set to zero. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):
2023.01.05 07:43:05.871276 [ 1688 ] {} <Error> void DB::MergeTreeBackgroundExecutor<DB::MergeMutateRuntimeQueue>::routine(DB::TaskRuntimeDataPtr) [Queue = DB::MergeMutateRuntimeQueue]: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 34.33 GiB (attempt to allocate chunk of 1051048 bytes), maximum: 34.29 GiB. OvercommitTracker decision: Memory overcommit isn't used. Waiting time or overcommit denominator are set to zero. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):
2023.01.05 07:43:05.876756 [ 1683 ] {} <Error> void DB::MergeTreeBackgroundExecutor<DB::MergeMutateRuntimeQueue>::routine(DB::TaskRuntimeDataPtr) [Queue = DB::MergeMutateRuntimeQueue]: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 34.33 GiB (attempt to allocate chunk of 1072930 bytes), maximum: 34.29 GiB. OvercommitTracker decision: Memory overcommit isn't used. Waiting time or overcommit denominator are set to zero. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):
2023.01.05 07:43:05.887665 [ 1674 ] {} <Error> void DB::MergeTreeBackgroundExecutor<DB::MergeMutateRuntimeQueue>::routine(DB::TaskRuntimeDataPtr) [Queue = DB::MergeMutateRuntimeQueue]: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 34.33 GiB (attempt to allocate chunk of 1051048 bytes), maximum: 34.29 GiB. OvercommitTracker decision: Memory overcommit isn't used. Waiting time or overcommit denominator are set to zero. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):
2023.01.05 07:43:05.893049 [ 1677 ] {} <Error> void DB::MergeTreeBackgroundExecutor<DB::MergeMutateRuntimeQueue>::routine(DB::TaskRuntimeDataPtr) [Queue = DB::MergeMutateRuntimeQueue]: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 34.33 GiB (attempt to allocate chunk of 1072930 bytes), maximum: 34.29 GiB. OvercommitTracker decision: Memory overcommit isn't used. Waiting time or overcommit denominator are set to zero. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):
2023.01.05 07:43:05.906151 [ 1686 ] {} <Error> void DB::MergeTreeBackgroundExecutor<DB::MergeMutateRuntimeQueue>::routine(DB::TaskRuntimeDataPtr) [Queue = DB::MergeMutateRuntimeQueue]: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 34.33 GiB (attempt to allocate chunk of 1051048 bytes), maximum: 34.29 GiB. OvercommitTracker decision: Memory overcommit isn't used. Waiting time or overcommit denominator are set to zero. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):
2023.01.05 07:43:05.911196 [ 1681 ] {} <Error> void DB::MergeTreeBackgroundExecutor<DB::MergeMutateRuntimeQueue>::routine(DB::TaskRuntimeDataPtr) [Queue = DB::MergeMutateRuntimeQueue]: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 34.33 GiB (attempt to allocate chunk of 1072930 bytes), maximum: 34.29 GiB. OvercommitTracker decision: Memory overcommit isn't used. Waiting time or overcommit denominator are set to zero. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

so I guess it's the "deadlock" in OvercommitTracker (#45236)