Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

storage: make input stream buffer size configurable #3412

Closed
jcsp opened this issue Jan 7, 2022 · 1 comment · Fixed by #3421
Closed

storage: make input stream buffer size configurable #3412

jcsp opened this issue Jan 7, 2022 · 1 comment · Fixed by #3421
Labels
area/cloud-storage Shadow indexing subsystem area/storage kind/enhance New feature or request

Comments

@jcsp
Copy link
Contributor

jcsp commented Jan 7, 2022

The default 128kiB is too high for systems with high partition counts or small segment sizes (therefore may segments). Each partition can have several segments open (especially during compaction, shadow indexing upload, multiple readers at different offsets).

Something like 16KiB is more appropriate when the partition count is high.

We should make this configurable, and possibly also auto-adjust the buffer size if the partition count is above a certain amount.

Related: #3398, in which the memory footprint of a partition is dominated by number_of_open_segments * input_stream_buffer_size. The safety check should probably be parametrized on the input stream size. This will remain true until the reader code is re-worked to avoid holding readers open for every segment.
@jcsp jcsp added kind/enhance New feature or request area/storage area/cloud-storage Shadow indexing subsystem labels Jan 7, 2022
jcsp added a commit to jcsp/redpanda that referenced this issue Jan 10, 2022
@jcsp
storage: consume config settings for read buffer size
d61329b 
Fixes: redpanda-data#3412

Signed-off-by: John Spray <jcs@vectorized.io>
@jcsp jcsp mentioned this issue Jan 10, 2022
Merged
jcsp added a commit to jcsp/redpanda that referenced this issue Jan 10, 2022
@jcsp
storage: consume config settings for read buffer size
e8fa580 
Fixes: redpanda-data#3412

Signed-off-by: John Spray <jcs@vectorized.io>
jcsp added a commit to jcsp/redpanda that referenced this issue Jan 10, 2022
@jcsp
storage: consume config settings for read buffer size
05e8c99 
Fixes: redpanda-data#3412

Signed-off-by: John Spray <jcs@vectorized.io>
jcsp added a commit to jcsp/redpanda that referenced this issue Jan 10, 2022
@jcsp
storage: consume config settings for read buffer size
b5ac363 
Fixes: redpanda-data#3412

Signed-off-by: John Spray <jcs@vectorized.io>
(cherry picked from commit 05e8c99)
jcsp added a commit to jcsp/redpanda that referenced this issue Jan 11, 2022
@jcsp
storage: consume config settings for read buffer size
56b2395 
Fixes: redpanda-data#3412

Signed-off-by: John Spray <jcs@vectorized.io>
(cherry picked from commit 05e8c99)
@jcsp
Copy link
Contributor Author

jcsp commented Jan 11, 2022 •
edited

Backporting to 21.11.x here:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/cloud-storage Shadow indexing subsystem area/storage kind/enhance New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

storage/cloud_storage: make read buffer sizes configurable jcsp/redpanda
1 participant
@jcsp