-
Notifications
You must be signed in to change notification settings - Fork 552
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changed the partition_status
chunked_fifo
chunk size to 8
#16247
Changed the partition_status
chunked_fifo
chunk size to 8
#16247
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM as a stop-gap until we have a better structure.
This will at least reduce memory use by ~16x for 1-partition topics, which should be enough to push the memory use down to a reasonable value (maybe ~100 MB ish).
We basically want a structure has the memory usage of |
14ae4c7
ca1ba12
to
14ae4c7
Compare
Still see some build failures |
Signed-off-by: Michal Maslanka <michal@redpanda.com>
Signed-off-by: Michal Maslanka <michal@redpanda.com>
With large number of topics that contain a small amount of partitions we may end up wasting a lot of memory using the default `ss::chunked_fifo` chunk size of 128. Change the chunk size of `partition_status` `chunked_fifo` collection to 8 to minimize a waste with large number of small topics. Signed-off-by: Michal Maslanka <michal@redpanda.com>
75c9549
14ae4c7
to
75c9549
Compare
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/44200#018d3aca-a16c-4738-b7e9-91579f903e8f |
/backport v23.3.x |
/backport v23.2.x |
With large number of topics that contain a small amount of partitions we
may end up wasting a lot of memory using the default
ss::chunked_fifo
chunk size of 128.
Change the chunk size of
partition_status
chunked_fifo
collection to8 to minimize a waste with large number of small topics.
Backports Required
Release Notes
Improvements