cdc: ensure that CDC generation write is flushed to commitlog before ack #7619
Conversation
When a node bootstraps or upgrades from a pre-CDC version, it creates a new CDC generation, writes it to a distributed table (system_distributed.cdc_generation_descriptions), and starts gossiping its timestamp. When other nodes see the timestamp being gossiped, they retrieve the generation from the table. The bootstrapping/upgrading node therefore assumes that the generation is made durable and other nodes will be able to retrieve it from the table. This assumption could be invalidated if periodic commitlog mode was used: replicas would acknowledge the write and then immediately crash, losing the write if they were unlucky (i.e. commitlog wasn't synced to disk before the write was acknowledged). This commit enforces all writes to the generations table to be synced to commitlog immediately. It does not matter for performance as these writes are very rare. Fixes scylladb#7610.
|
Have you checked the dtest stops failing with this fix @kbr- ? |
|
Yes. I added a paragraph at the end of the PR letter which mentions it (refresh github if it doesn't show it). |
There was a problem hiding this comment.
Looks good to me. I'll commit.
@gleb-cloudius I'm worried that this is a slippery slope - that no that "extra durability" is available, everyone would want it. Who will settle with "durability" when "extra durability" is available? And this is an example. But I understood during today's status call (I hope I'm not mistaken) that Gleb is aware of this new use and aproves of it.
|
Don't you need to add a call to |
|
@gleb-cloudius it looks like it's not needed, the |
When a node bootstraps or upgrades from a pre-CDC version, it creates a new CDC generation, writes it to a distributed table (system_distributed.cdc_generation_descriptions), and starts gossiping its timestamp. When other nodes see the timestamp being gossiped, they retrieve the generation from the table. The bootstrapping/upgrading node therefore assumes that the generation is made durable and other nodes will be able to retrieve it from the table. This assumption could be invalidated if periodic commitlog mode was used: replicas would acknowledge the write and then immediately crash, losing the write if they were unlucky (i.e. commitlog wasn't synced to disk before the write was acknowledged). This commit enforces all writes to the generations table to be synced to commitlog immediately. It does not matter for performance as these writes are very rare. Fixes #7610. Closes #7619 (cherry picked from commit d74f303)
|
Backported to 4.3. |
|
@avikivity could we also backport this to 4.2? |
When a node bootstraps or upgrades from a pre-CDC version, it creates a
new CDC generation, writes it to a distributed table
(
system_distributed.cdc_generation_descriptions), and starts gossipingits timestamp. When other nodes see the timestamp being gossiped, they
retrieve the generation from the table.
The bootstrapping/upgrading node therefore assumes that the generation
is made durable and other nodes will be able to retrieve it from the
table. This assumption could be invalidated if periodic commitlog mode
was used: replicas would acknowledge the write and then immediately
crash, losing the write if they were unlucky (i.e. commitlog wasn't
synced to disk before the write was acknowledged).
This commit enforces all writes to the generations table to be
synced to commitlog immediately. It does not matter for performance as
these writes are very rare.
Fixes #7610.
Tested by inserting a tactical sleep in commitlog code which allowed me to reproduce issue #7610 with each run of the
test_periodic_commitlogtest. With this commit the bug no longer occurred.