Tracking: Direct CDC Source Enhancement

[StrikeW]

• Support lock-free and incremental snapshot RFC: CDC Source with Backfill rfcs#63

  • Support in-memory version of backfill for a single table - lock-free feat(cdc): in-memory backfill for single mysql cdc table  #11707
  • Persist backfill progress to support incremental backfill - incremental snapshot (Persist the backfill state to record backfill progress #13204)

• Share an upstream change log stream among multiple CDC tables

  • Support create a Source streaming job for cdc connector to forward upstream events feat(cdc): share a changelog stream for multiple cdc tables #12535
  • Derive query plan with Chain for the CREATE TABLE ... FROM ... command feat(cdc): share a changelog stream for multiple cdc tables #12535
  • Persist the backfill state to record backfill progress #13204
    • Add test for recoverable cdc backfill #13400
  • Make the data dispatch more elegant of tables that sharing a CDC source #13147
  • Support the functionality for more upstream (v1.6), e.g. postgres-cdc
    • feat(cdc): support postgres cdc backfill #13958

• Preserve upstream transactional semantics

  • feat(cdc): support transaction for shared cdc source #14375

• Observability

  • Report CDC connector error to grafana #14353

• Optimization

  • Optimize the snapshot read of CDC backfill #15812

• Misc

  • bug: altering table column failed when table is built upon a shared cdc source #16287