feat(connector): support transaction for single-table CDC #11453

BugenZhao · 2023-08-04T11:27:57Z

Signed-off-by: Bugen Zhao i@bugenzhao.comI hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

Support for handling transaction boundaries in Debezium CDC messages. Users can specify transactional = 'true' in the table definition to enable this feature. Tested locally.

The idea is relatively simple:

do not yield a chunk if we're currently in a transaction
if we're delayed by a transaction, yield it immediately after the current transaction is committed
if a transaction is too large, force commit

Checklist

I have written necessary rustdoc comments
I have added necessary unit tests and integration tests
All checks passed in ./risedev check (or alias, ./risedev c)

Documentation

My PR contains user-facing changes.

Click here for Documentation

Types of user-facing changes

Please keep the types that apply to your changes, and remove the others.

Connector (sources & sinks)

Release note

EXPERIMENTAL Specify transactional = 'true' in the WITH clause of a CDC table definition to provide atomicity for transactions in the upstream for this single table. For the consideration of performance, transactions involving changes of more than 4096 rows cannot be guaranteed.

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

codecov · 2023-08-07T08:57:13Z

Codecov Report

Merging #11453 (6243f35) into main (f76cbc5) will decrease coverage by 0.02%.
Report is 15 commits behind head on main.
The diff coverage is 18.33%.

@@            Coverage Diff             @@
##             main   #11453      +/-   ##
==========================================
- Coverage   70.42%   70.41%   -0.02%     
==========================================
  Files        1351     1351              
  Lines      225283   225338      +55     
==========================================
+ Hits       158655   158664       +9     
- Misses      66628    66674      +46

Flag	Coverage Δ
rust	`70.41% <18.33%> (-0.02%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
src/connector/src/parser/mod.rs	`45.60% <3.70%> (-3.07%)`	⬇️
src/connector/src/parser/unified/debezium.rs	`58.09% <17.64%> (-7.82%)`	⬇️
...c/connector/src/parser/debezium/debezium_parser.rs	`51.25% <40.00%> (-3.17%)`	⬇️
src/connector/src/parser/unified/mod.rs	`87.50% <100.00%> (+0.83%)`	⬆️

... and 2 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

fuyufjh

The rest LGTM

fuyufjh · 2023-08-08T14:18:27Z

src/connector/src/parser/mod.rs

-            split_offset_mapping: Some(split_offset_mapping),
-        };
+        // If we are not in a transaction, we should yield the chunk now.
+        if current_transaction.is_none() {


Shall we limit the max size of a transaction? Similar with the limitation for DML transaction. Otherwise, it's possible that a very large transaction from upstream DB causes RW to crash

risingwave/src/stream/src/executor/dml.rs

Lines 57 to 64 in 0a9d2b4

/// If a transaction's data is less than `MAX_CHUNK_FOR_ATOMICITY` * `CHUNK_SIZE`, we can provide

/// atomicity. Otherwise, it is possible that part of transaction's data is sent to the downstream

/// without barrier boundaries. There are some cases that could cause non-atomicity for large

/// transaction. 1. The system crashes.

/// 2. Actor scale-in or migration.

/// 3. Dml's batch query error occurs at the middle of its execution. (e.g. Remove UDF function

/// server become unavailable).

const MAX_CHUNK_FOR_ATOMICITY: usize = 32;

Done. PTAL 🥰

I think we need to unify these 2 numbers and document it somewhere in our docs. cc. @chenzl25

+1. We need to document it in DML and table CDC section.

@chenzl25 @fuyufjh @BugenZhao Please see this doc PR for the related doc updates. Please review and make edits as you see fit. Thanks.

tabVersion

basically LGTM

so we cannot handle case that txn and scale occur at the same time? and how to prevent it?

tabVersion · 2023-08-08T16:30:30Z

src/connector/src/parser/debezium/debezium_parser.rs

+    #[allow(clippy::unused_async)] // false positive for `async_trait`
    async fn parse_one<'a>(
+        &'a mut self,
+        _key: Option<Vec<u8>>,
+        _payload: Option<Vec<u8>>,
+        _writer: SourceStreamChunkRowWriter<'a>,
+    ) -> Result<WriteGuard> {
+        unreachable!("should call `parse_one_with_txn` instead")
+    }


just remove it?

It's a required method of this trait, used for simplifying the implementation of other connectors that do not care about transactions.

StrikeW

Rest lgtm. I think we should set a maximum events limit for transactions to prevent OOM.

src/connector/src/parser/mod.rs

BugenZhao · 2023-08-09T06:15:21Z

so we cannot handle case that txn and scale occur at the same time? and how to prevent it?

I don't quite get it. 👀 As we never yield a data chunk when currently inside a transaction, the offset persisted by the source executor for recovery and scaling will also respect the transaction. 🤔

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

BugenZhao added 3 commits August 4, 2023 18:57

initial support

327e6fb

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

refactor

25e1a77

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

fix clippy

9d5359f

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

github-actions bot added the type/feature label Aug 4, 2023

BugenZhao added 2 commits August 7, 2023 15:05

add docs

c54316c

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

property and refinement

65f23a4

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

BugenZhao requested review from StrikeW, fuyufjh, tabVersion and wugouzi August 7, 2023 07:25

BugenZhao marked this pull request as ready for review August 7, 2023 07:25

github-actions bot added the user-facing-changes Contains changes that are visible to users label Aug 7, 2023

Merge branch 'main' into bz/single-table-txn

7ce103b

Merge branch 'main' into bz/single-table-txn

f3991cd

fuyufjh approved these changes Aug 8, 2023

View reviewed changes

tabVersion approved these changes Aug 8, 2023

View reviewed changes

StrikeW approved these changes Aug 9, 2023

View reviewed changes

src/connector/src/parser/mod.rs Show resolved Hide resolved

limit transaction size

6243f35

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

BugenZhao added this pull request to the merge queue Aug 10, 2023

Merged via the queue into main with commit 31e68e9 Aug 10, 2023
39 checks passed

BugenZhao deleted the bz/single-table-txn branch August 10, 2023 06:47

This was referenced Sep 5, 2023

document the support for transactions in single-table CDC risingwavelabs/risingwave-docs#1256

Closed

document transactions within a CDC table risingwavelabs/risingwave-docs#1279

Merged

StrikeW mentioned this pull request Jan 3, 2024

Support transaction for shared cdc source #14348

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(connector): support transaction for single-table CDC #11453

feat(connector): support transaction for single-table CDC #11453

BugenZhao commented Aug 4, 2023 •

edited

Loading

codecov bot commented Aug 7, 2023 •

edited

Loading

fuyufjh left a comment

fuyufjh Aug 8, 2023

BugenZhao Aug 9, 2023

fuyufjh Aug 10, 2023

chenzl25 Aug 11, 2023

hengm3467 Sep 8, 2023

tabVersion left a comment

tabVersion Aug 8, 2023

BugenZhao Aug 9, 2023

StrikeW left a comment

BugenZhao commented Aug 9, 2023

	/// If a transaction's data is less than `MAX_CHUNK_FOR_ATOMICITY` * `CHUNK_SIZE`, we can provide
	/// atomicity. Otherwise, it is possible that part of transaction's data is sent to the downstream
	/// without barrier boundaries. There are some cases that could cause non-atomicity for large
	/// transaction. 1. The system crashes.
	/// 2. Actor scale-in or migration.
	/// 3. Dml's batch query error occurs at the middle of its execution. (e.g. Remove UDF function
	/// server become unavailable).
	const MAX_CHUNK_FOR_ATOMICITY: usize = 32;

feat(connector): support transaction for single-table CDC #11453

feat(connector): support transaction for single-table CDC #11453

Conversation

BugenZhao commented Aug 4, 2023 • edited Loading

What's changed and what's your intention?

Checklist

Documentation

Types of user-facing changes

Release note

codecov bot commented Aug 7, 2023 • edited Loading

Codecov Report

fuyufjh left a comment

Choose a reason for hiding this comment

fuyufjh Aug 8, 2023

Choose a reason for hiding this comment

BugenZhao Aug 9, 2023

Choose a reason for hiding this comment

fuyufjh Aug 10, 2023

Choose a reason for hiding this comment

chenzl25 Aug 11, 2023

Choose a reason for hiding this comment

hengm3467 Sep 8, 2023

Choose a reason for hiding this comment

tabVersion left a comment

Choose a reason for hiding this comment

tabVersion Aug 8, 2023

Choose a reason for hiding this comment

BugenZhao Aug 9, 2023

Choose a reason for hiding this comment

StrikeW left a comment

Choose a reason for hiding this comment

BugenZhao commented Aug 9, 2023

BugenZhao commented Aug 4, 2023 •

edited

Loading

codecov bot commented Aug 7, 2023 •

edited

Loading