Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kafka: chunked_vector for config responses #17245

Merged
merged 7 commits into from
Mar 25, 2024
Merged

kafka: chunked_vector for config responses #17245

merged 7 commits into from
Mar 25, 2024

Conversation

pgellert
Copy link
Contributor

This updates the CreateTopics and DescribeConfigs kafka api handlers to use chunked_vector instead of std::vector for the creation of their responses. This is to avoid fragmentation-related issues when the number of topics or partitions is large and so large allocations are necessary to generate the response.

Closes https://github.com/redpanda-data/core-internal/issues/1174

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v23.3.x
  • v23.2.x

Release Notes

  • none

@pgellert
kafka: generalize response writer container
6a5ceb8 
This is necessary to allow using chunked_vector with these encoder
methods.

This continues the work started in the following reference PR:
redpanda-data#8469
@pgellert
kafka: avoid copy on create topics response
1c114cd 
This changes two cases where the topics response was copied but instead
could be moved.

This is necessary to enable us to switch to using chunked_vector instead
of std::vector, which only supports move semantics, not copy.
@pgellert pgellert self-assigned this Mar 21, 2024
dotnwat
dotnwat previously approved these changes Mar 21, 2024
View reviewed changes
@pgellert pgellert force-pushed the perf/configs-chunked-vector branch from 70efe49 to 97ba86b Compare March 21, 2024 18:30
@dotnwat
Copy link
Member

dotnwat commented Mar 21, 2024 

Checking 52 includes from 9 modules against 52 exported files
Traceback (most recent call last):
  File "/home/runner/work/redpanda/redpanda/tools/lint-includes", line 174, in <module>
    run(root)
  File "/home/runner/work/redpanda/redpanda/tools/lint-includes", line 167, in run
    assert not violations, f"Non-exported include found: {violations}"
AssertionError: Non-exported include found: {PosixPath('container/include/container/fragmented_vector.h')}

Looking. Could be a bug in the script

@pgellert
Copy link
Contributor Author

Force-pushed to fix the includes highlighted by the linter

@dotnwat
Copy link
Member

dotnwat commented Mar 21, 2024

Oh cool, thanks. Looks like the linter was working correctly!

dotnwat
dotnwat previously approved these changes Mar 21, 2024
View reviewed changes
@pgellert
Copy link
Contributor Author

Yep, very useful 🙂

@pgellert pgellert force-pushed the perf/configs-chunked-vector branch from 97ba86b to 1042d9d Compare March 22, 2024 16:41
@pgellert
Copy link
Contributor Author

Force-pushed to make the unit tests compile

@pgellert pgellert requested a review from dotnwat March 22, 2024 16:42
@vbotbuildovich
Copy link
Collaborator

vbotbuildovich commented Mar 22, 2024 •
edited

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/46626#018e673a-668d-4551-87ab-bd6b94a76846

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/46626#018e673a-668a-4fa1-a6d7-ecef0afdcd49

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/46626#018e674c-1f4d-4e4d-9b74-e93aa0b35544

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/46670#018e6ce0-87e4-447f-8435-76583d490d66

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/46670#018e6cf3-5abf-48be-bbf8-e9b10b8b7747

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/46670#018e6cf3-5abd-46aa-af16-12c110dd221d

@dotnwat
Copy link
Member

dotnwat commented Mar 23, 2024

/ci-repeat 1

@pgellert pgellert merged commit be3c604 into redpanda-data:dev Mar 25, 2024
18 checks passed
@michael-redpanda
Copy link
Contributor

/backport v23.3.x

@vbotbuildovich vbotbuildovich mentioned this pull request Apr 10, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dotnwat dotnwat dotnwat approved these changes

@BenPope BenPope Awaiting requested review from BenPope

@michael-redpanda michael-redpanda Awaiting requested review from michael-redpanda

Assignees

@pgellert pgellert

Labels
area/redpanda
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@pgellert @dotnwat @vbotbuildovich @michael-redpanda