Fixed large allocation in kafka::wait_for_leaders
#16287
Conversation
|
/dt |
|
/ci-repeat 10 |
|
/ci-repeat 10 |
1 similar comment
|
/ci-repeat 10 |
mmaslankaprv
force-pushed
the
wait-for-leaders-large-alloc
branch
from
20666d7
to
204e1f5
Compare
|
/ci-repeat 10 |
|
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/44441#018d5671-7d76-49d7-a337-d78544fb0815 ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/44441#018d5671-7d75-45c1-983f-293bda48ad87 ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/44441#018d5671-7d77-4000-99ae-f3f7f04f915f ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/44441#018d5671-7d7b-496a-bbdb-3eaa08be2e48 ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/44441#018d5671-7d7a-447e-a2d4-d765dc5ac918 ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/44441#018d5671-7d7b-4c3f-b1a8-5126d06818d0 ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/44441#018d5671-7d7c-40cf-8ab0-812dbfb4ac7c |
|
/dt |
mmaslankaprv
force-pushed
the
wait-for-leaders-large-alloc
branch
from
204e1f5
to
ea86488
Compare
mmaslankaprv
force-pushed
the
wait-for-leaders-large-alloc
branch
from
ea86488
to
6a75ebc
Compare
Previously we used a simple `std::vector` of futures to make waiting for partition leaders concurrent. Using a vector has a drawback when dealing with large number of topics and partitions since it may be required to allocate large contiguous chunk of memory for a future vector. In this particular case we may not use a fragmented vector or chunked fifo as the `when_all` uses a plain vector internally. To make sure no large chunk of memory is allocated to wait for the partition leaders changed the logic to use `seastar::max_concurrent_for_each`. Fixes: redpanda-data#15908 Signed-off-by: Michal Maslanka <michal@redpanda.com>
Sometimes it may happen that producer swarm is stopped after topic is recreated leading to a test failure. Added check restarting the producer if necessary Signed-off-by: Michal Maslanka <michal@redpanda.com>
mmaslankaprv
force-pushed
the
wait-for-leaders-large-alloc
branch
from
6a75ebc
to
a18fdcd
Compare
|
/backport v23.3.x |
|
/backport v23.2.x |
|
Oops! Something went wrong. |
|
Oops! Something went wrong. |
|
/backport v23.3.x |
|
/backport v23.2.x |
Previously we used a simple
std::vectorof futures to makewaiting for partition leaders concurrent. Using a vector has a drawback
when dealing with large number of topics and partitions since it may be
required to allocate large contiguous chunk of memory for a future
vector. In this particular case we may not use a fragmented vector or
chunked fifo as the
when_alluses a plain vector internally.To make sure no large chunk of memory is allocated to wait for the
partition leaders changed the logic to use
seastar::max_concurrent_for_each.Fixes: #15908, #16270
Fixes: #16036
Backports Required
Release Notes