feat(cdc): in-memory backfill for single mysql cdc table #11707

StrikeW · 2023-08-16T06:09:15Z

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

This PR is the first part of the risingwavelabs/rfcs#63. It implements a CdcBackfillExecutor to take care of the backfill logic for a CDC table. Right now, we will instantiate a CdcBackfillExecutor to wrap a source executor so that we can use the changelog events to backfill the table.

Main changes are as follows:

Add a session variable cdc_backfill to gate the this new feature
Add a StreamChunkMeta field to the StreamChunk for storing offsets of change log events, because the backfill algorithm needs to filter events based on offset (snapshot chunk doesn't have offsets)
ExternalTableReader component for reading snapshot and binlog from upstream table, this PR implements a table reader for MySQL
Frontend will store external table information as a field in StreamSourceInfo in the process of CREATE TABLE

TODO:

This PR supports backfill for the MySQL table, Postgres support will be done in another PR
Records backfill offset so that we can resume the backfill progress upon recovery
~~- Rewrite primary key comparison conditions for MySQL~~
~~- Support more MySQL data types~~

tracked: #11079

Checklist

I have written necessary rustdoc comments
I have added necessary unit tests and integration tests
I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
All checks passed in ./risedev check (or alias, ./risedev c)
My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

Enhance the mysql-cdc connector to allow lock-free ingestion, the connector won't need to lock upstream tables.
set cdc_backfill="true" to enable this feature.

…labs/risingwave into siyuan/cdc-backfill-framework

…ramework-single-table

src/connector/src/source/external.rs

hzxa21 · 2023-08-30T03:43:59Z

src/connector/src/source/external.rs

+                        };
+                        (pk.clone(), val)
+                    } else {
+                        (pk.clone(), Value::NULL)


This will generate filter conditions like a > NULL or a = NULL, which are always evaluated to false. Is this expected?

Since primary key cannot be null in mysql, I fixed by returning an error to the caller.

hzxa21 · 2023-08-30T03:50:07Z

src/stream/src/executor/backfill/upstream_table/snapshot.rs

+}
+
+/// A wrapper of upstream table for snapshot read
+/// because we need to customize the snapshot read for managed upstream table (e.g. mv, index)


UpstreamTableReader is only used for cdc backfill. Based on the doc here, are we planning to use the same trait for cdc backfill and table/mv backfill?

Ideally, I want to unify the implementation of backfill for mv and backfill for external table. But the fact is that it is a little hard to tune the implementation details to become a unified backfill executor. May revisit this part in future.

hzxa21 · 2023-08-30T04:03:52Z

src/stream/src/executor/backfill/no_shuffle_backfill.rs

@@ -506,10 +506,11 @@ where
            )
            .await?;


nits: we can use map to call KeyedRow::into_owned_row to avoid introdcuing the owned_row_iter method.

hzxa21 · 2023-08-30T04:04:00Z

src/stream/src/executor/backfill/arrangement_backfill.rs

@@ -540,7 +540,7 @@ where
                .await?;


nits: we can use map to call KeyedRow::into_owned_row to avoid introdcuing the owned_row_iter method.

BugenZhao · 2023-08-30T04:09:05Z

src/connector/src/source/external.rs

+        let order_key = primary_keys.iter().join(",");
+        let sql = if start_pk.is_none() {
+            format!(
+                "SELECT {} FROM {} ORDER BY {}",


Could there be potential injection attacks here?

Could there be potential injection attacks here?

Are spaces allowed in the RW field name?

It seems possible.

dev=> create table t ("1; DROP DATABASE foo; SELECT 1" int); CREATE_TABLE dev=> select * from t; 1; DROP DATABASE foo; SELECT 1 -------------------------------- (0 rows)

Similar issues: #4217

Thanks for the reminder🥵

I will unify to binary protocol in a separate PR.

…ramework-single-table

kwannoel · 2023-08-31T04:02:16Z

Add a session variable cdc_backfill to gate the this new feature

Seems like it is set to true by default https://github.com/risingwavelabs/risingwave/pull/11707/files#diff-4e19861816f5cf913890db3abd8088d01fa135d85f43080d5e6ee5c6c8ac40aaR337

Is that expected?

StrikeW · 2023-08-31T04:10:40Z

Add a session variable cdc_backfill to gate the this new feature

Seems like it is set to true by default https://github.com/risingwavelabs/risingwave/pull/11707/files#diff-4e19861816f5cf913890db3abd8088d01fa135d85f43080d5e6ee5c6c8ac40aaR337

Is that expected?

Yes.

Co-authored-by: StrikeW <StrikeW@users.noreply.github.com>

StrikeW added 30 commits July 20, 2023 17:56

save work

5a85049

upstream table reader for backfill

3784dc4

backfill: put upstream table related file to a folder

b6d8c8c

save work

fd55342

save demo for mysql_async

643b00a

mysql external table reader

5b91a0a

convert mysql row to rw row

cf42f8f

move external table reader to connector crate

00575e7

introduce external table config

9607b0c

impl read current binlog offset

6f31e04

integrate external table reader into UpstreamTableReader

09bb759

save work

3f91655

cdc: add event offset to the stream chunk

c957a35

todo: handle chunk hash dispatch

c19cbc2

forward offsets in hash dispatcher

fb33beb

save work

656a426

implement cdc backfill (WIP)

604fc0c

save work

a0a2fb3

fix compile errors

769e72b

(WIP) add integration test

31e642b

save integration test

0a743c8

Merge branch 'siyuan/cdc-backfill-framework' of github.com:risingwave…

7f524f1

…labs/risingwave into siyuan/cdc-backfill-framework

add integration test

215206e

(WIP) update frondend for create CDC table

23c1ad8

study meta fragment

7ad6c6f

support backfill for single cdc table

72cf628

only enable backfill for the parallelism has cdc split

6405d37

select_with_strategy seems does not consume left_stream first

e204232

peek changelog upstream first before backfill

653a55e

clean code

cdbc6c2

StrikeW added 6 commits August 29, 2023 19:08

Merge remote-tracking branch 'origin/main' into siyuan/cdc-backfill-f…

744e7f1

…ramework-single-table

fix ut

3ed80bb

convert keyed_row to owned_row

ab6d702

fix logging level

78cd99c

fix sql select list

040582b

Merge remote-tracking branch 'origin/main' into siyuan/cdc-backfill-f…

05d3de8

…ramework-single-table

StrikeW force-pushed the siyuan/cdc-backfill-framework-single-table branch from e7b76ff to 05d3de8 Compare August 30, 2023 03:10

hzxa21 reviewed Aug 30, 2023

View reviewed changes

BugenZhao reviewed Aug 30, 2023

View reviewed changes

StrikeW added 3 commits August 30, 2023 13:11

persist binlog offset when backfill finished

dc3ca09

fix clippy

0893f8e

error handling for null pk column & unsupported data type

1280535

StrikeW enabled auto-merge August 30, 2023 07:34

StrikeW added 2 commits August 30, 2023 15:55

rename file

8f196f2

fix clippy

bb0e705

StrikeW added this pull request to the merge queue Aug 30, 2023

Merge remote-tracking branch 'origin/main' into siyuan/cdc-backfill-f…

857df73

…ramework-single-table

StrikeW removed this pull request from the merge queue due to a manual request Aug 30, 2023

StrikeW enabled auto-merge August 30, 2023 09:42

StrikeW added this pull request to the merge queue Aug 30, 2023

Merged via the queue into main with commit 15f4fdd Aug 30, 2023
36 of 37 checks passed

StrikeW deleted the siyuan/cdc-backfill-framework-single-table branch August 30, 2023 11:39

This was referenced Sep 4, 2023

update cdc docs for v1.2 and previous versions risingwavelabs/risingwave-docs#1241

Merged

Require LOCK TABLES privilege in MySQL CDC connector #11906

Closed

Add description to highlight the CDC backfill new feature risingwavelabs/risingwave-docs#1244

Closed

xiangjinwu mentioned this pull request Sep 6, 2023

fix(expr): parse timestamptz without seconds but with offset #12084

Merged

8 tasks

Li0k pushed a commit that referenced this pull request Sep 15, 2023

feat(cdc): in-memory backfill for single mysql cdc table (#11707)

18ea4e8

Co-authored-by: StrikeW <StrikeW@users.noreply.github.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(cdc): in-memory backfill for single mysql cdc table #11707

feat(cdc): in-memory backfill for single mysql cdc table #11707

StrikeW commented Aug 16, 2023 •

edited

hzxa21 Aug 30, 2023

StrikeW Aug 30, 2023

hzxa21 Aug 30, 2023

StrikeW Aug 30, 2023

hzxa21 Aug 30, 2023

hzxa21 Aug 30, 2023

BugenZhao Aug 30, 2023

hzxa21 Aug 30, 2023 •

edited

BugenZhao Aug 30, 2023

StrikeW Aug 30, 2023

StrikeW Aug 30, 2023 •

edited

kwannoel commented Aug 31, 2023

StrikeW commented Aug 31, 2023

feat(cdc): in-memory backfill for single mysql cdc table #11707

feat(cdc): in-memory backfill for single mysql cdc table #11707

Conversation

StrikeW commented Aug 16, 2023 • edited

What's changed and what's your intention?

Checklist

Documentation

Release note

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hzxa21 Aug 30, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

StrikeW Aug 30, 2023 • edited

Choose a reason for hiding this comment

kwannoel commented Aug 31, 2023

StrikeW commented Aug 31, 2023

StrikeW commented Aug 16, 2023 •

edited

hzxa21 Aug 30, 2023 •

edited

StrikeW Aug 30, 2023 •

edited