-
Notifications
You must be signed in to change notification settings - Fork 3.7k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
changefeedccl: Reduce message duplicates during node restart
Changefeeds have at least once semantics. These semantics are implemnted via reliance on the closed timestamp system, which drives the changefeed checkpointing logic. When the node is restarted (gracefully) by draining the node, the changefeed will restart, re-emitting messages written since the last successfull checkpoint. During rolling cluster restarts, however, this behavior results in an almost quadratic behavior with respect to duplicate messages (e.g. a node may be drained every 5 minutes, and the checkpoint maybe produced every 5 minutes -- with each subsequent node being drained right before successfull checkpoint). This PR addresses the issue of duplicates during node (and cluster) restarts. First, this information is exposed via `OnDrain` channel made available in jobs registry. This channel is signaled when the node begins its drain process, and there is a wait period, configured via `server.shutdown.jobs_wait` setting -- 10 seconds by default -- before the registry will shut down, cancelling all currently running jobs. During this time, the changefeed aggregator running on the node being drained, detects this and shuts down -- transmitting its full frontier to the changfeed coordinator. Coordinator, in turn, sends signal to the remaining aggregators to exit -- also transmissintg their up-to-date frontier information. Prior to retrying the changefeed, an up-to-date frontier is reconstructed, persisted to the jobs table as needed, and the changefeed flow is replanned, avoiding the node that is being drained. Epic: CRDB-26978 Release note (enterprise change): Changefeeds emit significantly fewer duplicate messages during node/cluster restarts.
- Loading branch information
Yevgeniy Miretskiy
committed
May 8, 2023
1 parent
2973e83
commit 880aaf1
Showing
17 changed files
with
665 additions
and
132 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.