Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[v23.3.x] Refactored cluster::partition_leaders_table to use hierarchical structure of metadata #16709

Merged
merged 19 commits into from
Feb 27, 2024
Merged

[v23.3.x] Refactored cluster::partition_leaders_table to use hierarchical structure of metadata #16709

merged 19 commits into from
Feb 27, 2024

Conversation

mmaslankaprv
Copy link
Member

@mmaslankaprv mmaslankaprv commented Feb 26, 2024 •
edited
Loading

Backport of PR #16512
Backport of PR #16711

@mmaslankaprv
c/leaders_table: store leader information in hierarchical data structure
4fdd0ec 
Changed the way how leader information is stored in
`cluster::partition_leaders_table`. Previously each partition was
represented as an entry in leaders map. This way all partitions from the
same topic required a full `model::ntp` based lookup. Change the
implementation to the one where top level map is keyed by topic and its
value is a map storing leader per partition. This way all topic
partitions can be updated without the need to look up the whole topic.

Signed-off-by: Michal Maslanka <michal@redpanda.com>
(cherry picked from commit 2410d67)
@mmaslankaprv mmaslankaprv force-pushed the vbotbuildovich/backport-16512-v23.3.x-284 branch from c65d65c to 25353dd Compare February 26, 2024 11:28
Michal Maslanka and others added 16 commits February 26, 2024 12:43
store_leader_info
368025d 
Signed-off-by: Michal Maslanka <michal@redpadna.com>
@mmaslankaprv
c/leaders: track number of leader less partitions
4b2c02c 
Added tracking the number of leader less partitions in leaders table.
This prevents iterating over the whole list of leaders when generating
cluster metrics.

Signed-off-by: Michal Maslanka <michal@redpanda.com>
(cherry picked from commit 0d946e0)
@mmaslankaprv
c/controller_probe: do not iterate over all leaders
9bb5673 
Signed-off-by: Michal Maslanka <michal@redpanda.com>
(cherry picked from commit bd3cc30)
@mmaslankaprv
c/leaders: track version of leaders table
d2ac3c0 
Added tracking version of partition leaders table to be able to identify
concurrent modification. This will allow yielding while iterating the
leaders table. If a table is modified during operation an exception is
thrown and operation can be retried.

Signed-off-by: Michal Maslanka <michal@redpanda.com>
(cherry picked from commit b397956)
@mmaslankaprv
c/md_dissemiantion: replaced chunked fifo with new adaptive frag vec
290bd2c 
Signed-off-by: Michal Maslanka <michal@redpanda.com>
(cherry picked from commit abe6173)
@mmaslankaprv
model: added controller namespace topic definition
eb34dd7 
Signed-off-by: Michal Maslanka <michal@redpanda.com>
(cherry picked from commit fdeb6ac)
@mmaslankaprv
c/md_dissemination: optimize leader metadata update with health report
32b4ade 
Leveraging the hierarchical structure of node health report and
internals of partition leaders table to minimize the number of lookups
in leaders map.

Signed-off-by: Michal Maslanka <michal@redpanda.com>
(cherry picked from commit 9a255f0)
@mmaslankaprv
c/md_service: explicitly handle get_leadership failures
b2a3d90 
Previously `get_leadership_reply` did not contain any information about
the operation state, therefore it was impossible to propagate service
error to the client. Added a field indicating if response is successful.

The field allow us to explicitly handle errors like partition leaders
table concurrent modification.

Signed-off-by: Michal Maslanka <michal@redpanda.com>
(cherry picked from commit aeab006)
@mmaslankaprv
tests: added partitions leader table test
a2eff06 
Signed-off-by: Michal Maslanka <michal@redpanda.com>
(cherry picked from commit 2db09f9)
@mmaslankaprv
ntp_callbacks: allow notifying without the need of creating ntp
3f10cd8 
Signed-off-by: Michal Maslanka <michal@redpanda.com>
(cherry picked from commit cddc003)
@mmaslankaprv
test: added gtest assert_throw for coroutines
3ac99f3 
Signed-off-by: Michal Maslanka <michal@redpanda.com>
(cherry picked from commit e37df0c)
@mmaslankaprv
c/leaders_table: unify notification mechanism
d231f3f 
Using `ntp_callbacks` to wait for the leaders without additional
promises map. When caller requests to wait for a leader we register the
notification which sets the promise value when called. This way we do
not need a separate mechanism to keep track of leadership notifications.

Signed-off-by: Michal Maslanka <michal@redpanda.com>
(cherry picked from commit 4eed31f)
@mmaslankaprv
c/hm_types: replaced chunked_fifo with chunked_vector in health report
0f274ef 
Replaced previously used `chunked_fifo` with dynamically sized
fragmented vectors. Fragemented vector provides a random access iterator
and automatically controls the size of allocated chunks

Signed-off-by: Michal Maslanka <michal@redpanda.com>
(cherry picked from commit 44e6cd7)
@mmaslankaprv
container: added missing iterator methods to fragmented_vector
c3d7ba5 
Added methods allowing `fragmented_vector::iter` to satisfy
`std::random_random__iterator` concept.

Signed-off-by: Michal Maslanka <michal@redpanda.com>
(cherry picked from commit f4e6bce)
@mmaslankaprv
c/leaders_table: use async algorithm when updating leaders with report
c13a5a4 
Using async algorithm will call `ss::coroutine::maybe_yield()` every 100
operations while still being lightweight while iterating over
synchronously over a chunk.

Signed-off-by: Michal Maslanka <michal@redpanda.com>
(cherry picked from commit 9dbeb81)
@mmaslankaprv
c/leaders_table: use async algorithm to iterate over leaders
45466d5 
Signed-off-by: Michal Maslanka <michal@redpanda.com>
(cherry picked from commit 8948fe5)
@mmaslankaprv mmaslankaprv force-pushed the vbotbuildovich/backport-16512-v23.3.x-284 branch from 25353dd to 45466d5 Compare February 26, 2024 11:44
@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ mmaslankaprv
❌ Michal Maslanka

Michal Maslanka seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

1 similar comment
@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ mmaslankaprv
❌ Michal Maslanka

Michal Maslanka seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

@mmaslankaprv
c/p_leaders: check if leaders table was modified after async iteration
897470d 
Since leaders table might have been modified during the execution of
iteration over the topic partitions we must check the version of topics
table after the asynchronous iteration finished.

Signed-off-by: Michal Maslanka <michal@redpanda.com>
(cherry picked from commit 625fb44)
@mmaslankaprv
c/leaders_table: fixed logging skipped update
391fb46 
Signed-off-by: Michal Maslanka <michal@redpanda.com>
(cherry picked from commit 6eaa7b8)
@mmaslankaprv mmaslankaprv merged commit 05ed8e0 into redpanda-data:v23.3.x Feb 27, 2024
16 of 17 checks passed
@mmaslankaprv mmaslankaprv deleted the vbotbuildovich/backport-16512-v23.3.x-284 branch February 27, 2024 13:40
@mmaslankaprv mmaslankaprv mentioned this pull request Mar 8, 2024
Closed
@BenPope BenPope added this to the v23.3.7 milestone Mar 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@StephanDollberg StephanDollberg StephanDollberg approved these changes

Assignees
No one assigned
Labels
area/redpanda
Projects
None yet
Milestone
v23.3.7
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@mmaslankaprv @CLAassistant @StephanDollberg @BenPope