Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace topology::transition_state::publish_cdc_generation with a queue of generations to publish and bg fiber to do it #15194

Closed
gleb-cloudius opened this issue Aug 28, 2023 · 1 comment · Fixed by #15281
Assignees
Milestone

Comments

@gleb-cloudius
Copy link
Contributor

Currently topology coordinator has a topology::transition_state::publish_cdc_generation state where already created cdc generation is published to the user visible tables. This process cannot fail even if the bootstrap is aborted (otherwise some cdc updates will be missed)! With gossiper we have a fiber on each node that tries to publish the latest generation until it eventually succeed. With topology coordinator we have two options: either we hang in publish_cdc_generation state until it succeeds (this will block all other topology operations) or replace publish_cdc_generation with a static row in the topology state table that will hold not yet published generations. Instead of moving to publish_cdc_generation the generation will be added to the row. The topology coordinator will start a background fiber that will constantly try to publish all not yet published cdc generations atomically.

To publish cdc generation atomically system_distributed_keyspace::create_cdc_desc need to be changed to handle multiple generations and write multiple timestamps into cdc_generation_timestamps atomically.

@denesb
Copy link
Contributor

denesb commented Dec 18, 2023

I think CDC was not going through RAFT in 5.2 yet, so no backport needed. @kbr-scylla please re-add the label if I was wrong.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants