Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(cdc): in-memory backfill for single mysql cdc table #11707

Merged
merged 80 commits into from
Aug 30, 2023
Merged

feat(cdc): in-memory backfill for single mysql cdc table #11707

Show file tree
Hide file tree
Changes from 37 commits
Commits
Show all changes
80 commits
Select commit Hold shift + click to select a range
5a85049
save work
StrikeW Jul 20, 2023
3784dc4
upstream table reader for backfill
StrikeW Jul 20, 2023
b6d8c8c
backfill: put upstream table related file to a folder
StrikeW Jul 21, 2023
fd55342
save work
StrikeW Jul 21, 2023
643b00a
save demo for mysql_async
StrikeW Jul 24, 2023
5b91a0a
mysql external table reader
StrikeW Jul 25, 2023
cf42f8f
convert mysql row to rw row
StrikeW Jul 25, 2023
00575e7
move external table reader to connector crate
StrikeW Jul 25, 2023
9607b0c
introduce external table config
StrikeW Jul 26, 2023
6f31e04
impl read current binlog offset
StrikeW Jul 26, 2023
09bb759
integrate external table reader into UpstreamTableReader
StrikeW Jul 26, 2023
3f91655
save work
StrikeW Jul 26, 2023
c957a35
cdc: add event offset to the stream chunk
StrikeW Jul 27, 2023
c19cbc2
todo: handle chunk hash dispatch
StrikeW Jul 27, 2023
fb33beb
forward offsets in hash dispatcher
StrikeW Jul 28, 2023
656a426
save work
StrikeW Jul 28, 2023
604fc0c
implement cdc backfill (WIP)
StrikeW Jul 31, 2023
a0a2fb3
save work
StrikeW Jul 31, 2023
769e72b
fix compile errors
StrikeW Aug 2, 2023
31e642b
(WIP) add integration test
StrikeW Aug 2, 2023
0a743c8
save integration test
StrikeW Aug 3, 2023
7f524f1
Merge branch 'siyuan/cdc-backfill-framework' of github.com:risingwave…
StrikeW Aug 4, 2023
215206e
add integration test
StrikeW Aug 4, 2023
23c1ad8
(WIP) update frondend for create CDC table
StrikeW Aug 7, 2023
7ad6c6f
study meta fragment
StrikeW Aug 8, 2023
72cf628
support backfill for single cdc table
StrikeW Aug 9, 2023
6405d37
only enable backfill for the parallelism has cdc split
StrikeW Aug 10, 2023
e204232
select_with_strategy seems does not consume left_stream first
StrikeW Aug 11, 2023
653a55e
peek changelog upstream first before backfill
StrikeW Aug 14, 2023
cdbc6c2
clean code
StrikeW Aug 15, 2023
a777547
support scan from a given pk pos
StrikeW Aug 15, 2023
ba4d63d
minor
StrikeW Aug 16, 2023
109f855
minor
StrikeW Aug 16, 2023
1fdbcd3
fix sql that scan with given primary key
StrikeW Aug 16, 2023
7e2ebf3
clean code
StrikeW Aug 16, 2023
7e77861
minor
StrikeW Aug 16, 2023
9f00b1f
Merge remote-tracking branch 'origin/main' into siyuan/cdc-backfill-f…
StrikeW Aug 17, 2023
0004e99
minor
StrikeW Aug 17, 2023
485d5cd
save work
StrikeW Aug 17, 2023
c657eea
rewrite where filter expr for mysql
StrikeW Aug 18, 2023
4ea9a82
consume latest binlog
StrikeW Aug 22, 2023
13f8f56
add internal _rw_offset column to store offset
StrikeW Aug 23, 2023
3a7bbdd
minor
StrikeW Aug 23, 2023
d5c6ed0
rename
StrikeW Aug 23, 2023
b3742ab
remove is_cdc_table from Scan
StrikeW Aug 23, 2023
e7df3c9
minor
StrikeW Aug 23, 2023
51641b4
minor
StrikeW Aug 23, 2023
6f37dbb
minor
StrikeW Aug 23, 2023
dbcac38
minor
StrikeW Aug 23, 2023
28892e1
support more mysql data types
StrikeW Aug 24, 2023
a954c9f
Merge remote-tracking branch 'origin/main' into siyuan/cdc-backfill-f…
StrikeW Aug 24, 2023
f36b2c1
Fix "cargo-hakari"
StrikeW Aug 24, 2023
cccf9e7
minor
StrikeW Aug 24, 2023
7cb1fe8
fix cdc_backfill session variable
StrikeW Aug 24, 2023
8cb12ed
minor
StrikeW Aug 24, 2023
967b92b
fix data types
StrikeW Aug 24, 2023
eba646f
fix
StrikeW Aug 24, 2023
d84abea
enable cdc backfill on integration test
StrikeW Aug 24, 2023
f088e73
remove is_offset
StrikeW Aug 25, 2023
d41df96
refactor SourceColumnDesc
StrikeW Aug 27, 2023
f38c340
Merge remote-tracking branch 'origin/main' into siyuan/cdc-backfill-f…
StrikeW Aug 28, 2023
9a0d8dd
check last column must be _rw_offset
StrikeW Aug 28, 2023
8eed3cd
fix
StrikeW Aug 28, 2023
1f8a1df
add comments
StrikeW Aug 28, 2023
260e3d0
fix
StrikeW Aug 28, 2023
4e1ad92
persist backfill finish flag
StrikeW Aug 29, 2023
fa907bc
minor
StrikeW Aug 29, 2023
b1ace79
enable cdc backfill for mysql
StrikeW Aug 29, 2023
744e7f1
Merge remote-tracking branch 'origin/main' into siyuan/cdc-backfill-f…
StrikeW Aug 29, 2023
3ed80bb
fix ut
StrikeW Aug 29, 2023
ab6d702
convert keyed_row to owned_row
StrikeW Aug 29, 2023
78cd99c
fix logging level
StrikeW Aug 29, 2023
040582b
fix sql select list
StrikeW Aug 30, 2023
05d3de8
Merge remote-tracking branch 'origin/main' into siyuan/cdc-backfill-f…
StrikeW Aug 30, 2023
dc3ca09
persist binlog offset when backfill finished
StrikeW Aug 30, 2023
0893f8e
fix clippy
StrikeW Aug 30, 2023
1280535
error handling for null pk column & unsupported data type
StrikeW Aug 30, 2023
8f196f2
rename file
StrikeW Aug 30, 2023
bb0e705
fix clippy
StrikeW Aug 30, 2023
857df73
Merge remote-tracking branch 'origin/main' into siyuan/cdc-backfill-f…
StrikeW Aug 30, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
258 changes: 258 additions & 0 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading