Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(cdc): in-memory backfill for single mysql cdc table #11707

Merged
merged 80 commits into from
Aug 30, 2023
Merged

feat(cdc): in-memory backfill for single mysql cdc table #11707

merged 80 commits into from
Aug 30, 2023

Commits on Jul 20, 2023

  1. save work

    @StrikeW
    StrikeW committed Jul 20, 2023
    Configuration menu
    Copy the full SHA
    5a85049 View commit details
    Browse the repository at this point in the history

  2. upstream table reader for backfill

    @StrikeW
    StrikeW committed Jul 20, 2023
    Configuration menu
    Copy the full SHA
    3784dc4 View commit details
    Browse the repository at this point in the history

Commits on Jul 21, 2023

  1. backfill: put upstream table related file to a folder

    @StrikeW
    StrikeW committed Jul 21, 2023
    Configuration menu
    Copy the full SHA
    b6d8c8c View commit details
    Browse the repository at this point in the history

  2. save work

    @StrikeW
    StrikeW committed Jul 21, 2023
    Configuration menu
    Copy the full SHA
    fd55342 View commit details
    Browse the repository at this point in the history

Commits on Jul 24, 2023

  1. save demo for mysql_async

    @StrikeW
    StrikeW committed Jul 24, 2023
    Configuration menu
    Copy the full SHA
    643b00a View commit details
    Browse the repository at this point in the history

Commits on Jul 25, 2023

  1. mysql external table reader

    @StrikeW
    StrikeW committed Jul 25, 2023
    Configuration menu
    Copy the full SHA
    5b91a0a View commit details
    Browse the repository at this point in the history

  2. convert mysql row to rw row

    @StrikeW
    StrikeW committed Jul 25, 2023
    Configuration menu
    Copy the full SHA
    cf42f8f View commit details
    Browse the repository at this point in the history

  3. move external table reader to connector crate

    @StrikeW
    StrikeW committed Jul 25, 2023
    Configuration menu
    Copy the full SHA
    00575e7 View commit details
    Browse the repository at this point in the history

Commits on Jul 26, 2023

  1. introduce external table config

    @StrikeW
    StrikeW committed Jul 26, 2023
    Configuration menu
    Copy the full SHA
    9607b0c View commit details
    Browse the repository at this point in the history

  2. impl read current binlog offset

    @StrikeW
    StrikeW committed Jul 26, 2023
    Configuration menu
    Copy the full SHA
    6f31e04 View commit details
    Browse the repository at this point in the history

  3. integrate external table reader into UpstreamTableReader

    @StrikeW
    StrikeW committed Jul 26, 2023
    Configuration menu
    Copy the full SHA
    09bb759 View commit details
    Browse the repository at this point in the history

  4. save work

    @StrikeW
    StrikeW committed Jul 26, 2023
    Configuration menu
    Copy the full SHA
    3f91655 View commit details
    Browse the repository at this point in the history

Commits on Jul 27, 2023

  1. cdc: add event offset to the stream chunk

    @StrikeW
    StrikeW committed Jul 27, 2023
    Configuration menu
    Copy the full SHA
    c957a35 View commit details
    Browse the repository at this point in the history

  2. todo: handle chunk hash dispatch

    @StrikeW
    StrikeW committed Jul 27, 2023
    Configuration menu
    Copy the full SHA
    c19cbc2 View commit details
    Browse the repository at this point in the history

Commits on Jul 28, 2023

  1. forward offsets in hash dispatcher

    @StrikeW
    StrikeW committed Jul 28, 2023
    Configuration menu
    Copy the full SHA
    fb33beb View commit details
    Browse the repository at this point in the history

  2. save work

    @StrikeW
    StrikeW committed Jul 28, 2023
    Configuration menu
    Copy the full SHA
    656a426 View commit details
    Browse the repository at this point in the history

Commits on Jul 31, 2023

  1. implement cdc backfill (WIP)

    @StrikeW
    StrikeW committed Jul 31, 2023
    Configuration menu
    Copy the full SHA
    604fc0c View commit details
    Browse the repository at this point in the history

  2. save work

    @StrikeW
    StrikeW committed Jul 31, 2023
    Configuration menu
    Copy the full SHA
    a0a2fb3 View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2023

  1. fix compile errors

    @StrikeW
    StrikeW committed Aug 2, 2023
    Configuration menu
    Copy the full SHA
    769e72b View commit details
    Browse the repository at this point in the history

  2. (WIP) add integration test

    @StrikeW
    StrikeW committed Aug 2, 2023
    Configuration menu
    Copy the full SHA
    31e642b View commit details
    Browse the repository at this point in the history

Commits on Aug 3, 2023

  1. save integration test

    @StrikeW
    StrikeW committed Aug 3, 2023
    Configuration menu
    Copy the full SHA
    0a743c8 View commit details
    Browse the repository at this point in the history

Commits on Aug 4, 2023

  1. Merge branch 'siyuan/cdc-backfill-framework' of github.com:risingwave… 

    …labs/risingwave into siyuan/cdc-backfill-framework
    @StrikeW
    StrikeW committed Aug 4, 2023
    Configuration menu
    Copy the full SHA
    7f524f1 View commit details
    Browse the repository at this point in the history

Commits on Aug 6, 2023

  1. add integration test

    @StrikeW
    StrikeW committed Aug 6, 2023
    Configuration menu
    Copy the full SHA
    215206e View commit details
    Browse the repository at this point in the history

Commits on Aug 8, 2023

  1. (WIP) update frondend for create CDC table

    @StrikeW
    StrikeW committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    23c1ad8 View commit details
    Browse the repository at this point in the history

  2. study meta fragment

    @StrikeW
    StrikeW committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    7ad6c6f View commit details
    Browse the repository at this point in the history

Commits on Aug 9, 2023

  1. support backfill for single cdc table

    @StrikeW
    StrikeW committed Aug 9, 2023
    Configuration menu
    Copy the full SHA
    72cf628 View commit details
    Browse the repository at this point in the history

Commits on Aug 10, 2023

  1. only enable backfill for the parallelism has cdc split

    @StrikeW
    StrikeW committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    6405d37 View commit details
    Browse the repository at this point in the history

Commits on Aug 11, 2023

  1. select_with_strategy seems does not consume left_stream first

    @StrikeW
    StrikeW committed Aug 11, 2023
    Configuration menu
    Copy the full SHA
    e204232 View commit details
    Browse the repository at this point in the history

Commits on Aug 14, 2023

  1. peek changelog upstream first before backfill

    @StrikeW
    StrikeW committed Aug 14, 2023
    Configuration menu
    Copy the full SHA
    653a55e View commit details
    Browse the repository at this point in the history

Commits on Aug 15, 2023

  1. clean code

    @StrikeW
    StrikeW committed Aug 15, 2023
    Configuration menu
    Copy the full SHA
    cdbc6c2 View commit details
    Browse the repository at this point in the history

  2. support scan from a given pk pos

    @StrikeW
    StrikeW committed Aug 15, 2023
    Configuration menu
    Copy the full SHA
    a777547 View commit details
    Browse the repository at this point in the history

Commits on Aug 16, 2023

  1. minor

    @StrikeW
    StrikeW committed Aug 16, 2023
    Configuration menu
    Copy the full SHA
    ba4d63d View commit details
    Browse the repository at this point in the history

  2. minor

    @StrikeW
    StrikeW committed Aug 16, 2023
    Configuration menu
    Copy the full SHA
    109f855 View commit details
    Browse the repository at this point in the history

  3. fix sql that scan with given primary key

    @StrikeW
    StrikeW committed Aug 16, 2023
    Configuration menu
    Copy the full SHA
    1fdbcd3 View commit details
    Browse the repository at this point in the history

  4. clean code

    @StrikeW
    StrikeW committed Aug 16, 2023
    Configuration menu
    Copy the full SHA
    7e2ebf3 View commit details
    Browse the repository at this point in the history

  5. minor

    @StrikeW
    StrikeW committed Aug 16, 2023
    Configuration menu
    Copy the full SHA
    7e77861 View commit details
    Browse the repository at this point in the history

Commits on Aug 17, 2023

  1. Merge remote-tracking branch 'origin/main' into siyuan/cdc-backfill-f… 

    …ramework-single-table
    @StrikeW
    StrikeW committed Aug 17, 2023
    Configuration menu
    Copy the full SHA
    9f00b1f View commit details
    Browse the repository at this point in the history

  2. minor

    @StrikeW
    StrikeW committed Aug 17, 2023
    Configuration menu
    Copy the full SHA
    0004e99 View commit details
    Browse the repository at this point in the history

  3. save work

    @StrikeW
    StrikeW committed Aug 17, 2023
    Configuration menu
    Copy the full SHA
    485d5cd View commit details
    Browse the repository at this point in the history

Commits on Aug 18, 2023

  1. rewrite where filter expr for mysql

    @StrikeW
    StrikeW committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    c657eea View commit details
    Browse the repository at this point in the history

Commits on Aug 22, 2023

  1. consume latest binlog

    @StrikeW
    StrikeW committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    4ea9a82 View commit details
    Browse the repository at this point in the history

Commits on Aug 23, 2023

  1. add internal _rw_offset column to store offset

    @StrikeW
    StrikeW committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    13f8f56 View commit details
    Browse the repository at this point in the history

  2. minor

    @StrikeW
    StrikeW committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    3a7bbdd View commit details
    Browse the repository at this point in the history

  3. rename

    @StrikeW
    StrikeW committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    d5c6ed0 View commit details
    Browse the repository at this point in the history

  4. remove is_cdc_table from Scan

    @StrikeW
    StrikeW committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    b3742ab View commit details
    Browse the repository at this point in the history

  5. minor

    @StrikeW
    StrikeW committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    e7df3c9 View commit details
    Browse the repository at this point in the history

  6. minor

    @StrikeW
    StrikeW committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    51641b4 View commit details
    Browse the repository at this point in the history

  7. minor

    @StrikeW
    StrikeW committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    6f37dbb View commit details
    Browse the repository at this point in the history

  8. minor

    @StrikeW
    StrikeW committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    dbcac38 View commit details
    Browse the repository at this point in the history

Commits on Aug 24, 2023

  1. support more mysql data types

    @StrikeW
    StrikeW committed Aug 24, 2023
    Configuration menu
    Copy the full SHA
    28892e1 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/main' into siyuan/cdc-backfill-f… 

    …ramework-single-table
    @StrikeW
    StrikeW committed Aug 24, 2023
    Configuration menu
    Copy the full SHA
    a954c9f View commit details
    Browse the repository at this point in the history

  3. Fix "cargo-hakari"

    @StrikeW @github-actions
    StrikeW authored and github-actions[bot] committed Aug 24, 2023
    Configuration menu
    Copy the full SHA
    f36b2c1 View commit details
    Browse the repository at this point in the history

  4. minor

    @StrikeW
    StrikeW committed Aug 24, 2023
    Configuration menu
    Copy the full SHA
    cccf9e7 View commit details
    Browse the repository at this point in the history

  5. fix cdc_backfill session variable

    @StrikeW
    StrikeW committed Aug 24, 2023
    Configuration menu
    Copy the full SHA
    7cb1fe8 View commit details
    Browse the repository at this point in the history

  6. minor

    @StrikeW
    StrikeW committed Aug 24, 2023
    Configuration menu
    Copy the full SHA
    8cb12ed View commit details
    Browse the repository at this point in the history

  7. fix data types

    @StrikeW
    StrikeW committed Aug 24, 2023
    Configuration menu
    Copy the full SHA
    967b92b View commit details
    Browse the repository at this point in the history

  8. fix

    @StrikeW
    StrikeW committed Aug 24, 2023
    Configuration menu
    Copy the full SHA
    eba646f View commit details
    Browse the repository at this point in the history

  9. enable cdc backfill on integration test

    @StrikeW
    StrikeW committed Aug 24, 2023
    Configuration menu
    Copy the full SHA
    d84abea View commit details
    Browse the repository at this point in the history

Commits on Aug 25, 2023

  1. remove is_offset

    @StrikeW
    StrikeW committed Aug 25, 2023
    Configuration menu
    Copy the full SHA
    f088e73 View commit details
    Browse the repository at this point in the history

Commits on Aug 27, 2023

  1. refactor SourceColumnDesc

    @StrikeW
    StrikeW committed Aug 27, 2023
    Configuration menu
    Copy the full SHA
    d41df96 View commit details
    Browse the repository at this point in the history

Commits on Aug 28, 2023

  1. Merge remote-tracking branch 'origin/main' into siyuan/cdc-backfill-f… 

    …ramework-single-table
    @StrikeW
    StrikeW committed Aug 28, 2023
    Configuration menu
    Copy the full SHA
    f38c340 View commit details
    Browse the repository at this point in the history

  2. check last column must be _rw_offset

    @StrikeW
    StrikeW committed Aug 28, 2023
    Configuration menu
    Copy the full SHA
    9a0d8dd View commit details
    Browse the repository at this point in the history

  3. fix

    @StrikeW
    StrikeW committed Aug 28, 2023
    Configuration menu
    Copy the full SHA
    8eed3cd View commit details
    Browse the repository at this point in the history

  4. add comments

    @StrikeW
    StrikeW committed Aug 28, 2023
    Configuration menu
    Copy the full SHA
    1f8a1df View commit details
    Browse the repository at this point in the history

  5. fix

    @StrikeW
    StrikeW committed Aug 28, 2023
    Configuration menu
    Copy the full SHA
    260e3d0 View commit details
    Browse the repository at this point in the history

Commits on Aug 29, 2023

  1. persist backfill finish flag

    @StrikeW
    StrikeW committed Aug 29, 2023
    Configuration menu
    Copy the full SHA
    4e1ad92 View commit details
    Browse the repository at this point in the history

  2. minor

    @StrikeW
    StrikeW committed Aug 29, 2023
    Configuration menu
    Copy the full SHA
    fa907bc View commit details
    Browse the repository at this point in the history

  3. enable cdc backfill for mysql

    @StrikeW
    StrikeW committed Aug 29, 2023
    Configuration menu
    Copy the full SHA
    b1ace79 View commit details
    Browse the repository at this point in the history

  4. Merge remote-tracking branch 'origin/main' into siyuan/cdc-backfill-f… 

    …ramework-single-table
    @StrikeW
    StrikeW committed Aug 29, 2023
    Configuration menu
    Copy the full SHA
    744e7f1 View commit details
    Browse the repository at this point in the history

  5. fix ut

    @StrikeW
    StrikeW committed Aug 29, 2023
    Configuration menu
    Copy the full SHA
    3ed80bb View commit details
    Browse the repository at this point in the history

  6. convert keyed_row to owned_row

    @StrikeW
    StrikeW committed Aug 29, 2023
    Configuration menu
    Copy the full SHA
    ab6d702 View commit details
    Browse the repository at this point in the history

  7. fix logging level

    @StrikeW
    StrikeW committed Aug 29, 2023
    Configuration menu
    Copy the full SHA
    78cd99c View commit details
    Browse the repository at this point in the history

Commits on Aug 30, 2023

  1. fix sql select list

    @StrikeW
    StrikeW committed Aug 30, 2023
    Configuration menu
    Copy the full SHA
    040582b View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/main' into siyuan/cdc-backfill-f… 

    …ramework-single-table
    @StrikeW
    StrikeW committed Aug 30, 2023
    Configuration menu
    Copy the full SHA
    05d3de8 View commit details
    Browse the repository at this point in the history

  3. persist binlog offset when backfill finished

    @StrikeW
    StrikeW committed Aug 30, 2023
    Configuration menu
    Copy the full SHA
    dc3ca09 View commit details
    Browse the repository at this point in the history

  4. fix clippy

    @StrikeW
    StrikeW committed Aug 30, 2023
    Configuration menu
    Copy the full SHA
    0893f8e View commit details
    Browse the repository at this point in the history

  5. error handling for null pk column & unsupported data type

    @StrikeW
    StrikeW committed Aug 30, 2023
    Configuration menu
    Copy the full SHA
    1280535 View commit details
    Browse the repository at this point in the history

  6. rename file

    @StrikeW
    StrikeW committed Aug 30, 2023
    Configuration menu
    Copy the full SHA
    8f196f2 View commit details
    Browse the repository at this point in the history

  7. fix clippy

    @StrikeW
    StrikeW committed Aug 30, 2023
    Configuration menu
    Copy the full SHA
    bb0e705 View commit details
    Browse the repository at this point in the history

  8. Merge remote-tracking branch 'origin/main' into siyuan/cdc-backfill-f… 

    …ramework-single-table
    @StrikeW
    StrikeW committed Aug 30, 2023
    Configuration menu
    Copy the full SHA
    857df73 View commit details
    Browse the repository at this point in the history