Support sync in active-active mode to TiDB and sync active-active tables in non-active-active mode to all downstream (#3299)

[ti-chi-bot]

This is an automated cherry-pick of #3299

What problem does this PR solve?

Issue Number: close #3446

What is changed and how it works?

This pull request introduces foundational support for active-active replication in TiCDC, primarily for MySQL-compatible downstream systems like TiDB. It enables new configuration options for active-active mode, refines DML event processing to handle filtering and transformation based on replication policies, and establishes a mechanism for tracking replication progress in a dedicated table. These changes aim to enhance data consistency and conflict resolution capabilities in active-active environments.

Highlights

• Active-Active Replication Configuration: Introduced new configuration options, enable-active-active and active-active-progress-interval, within the ReplicaConfig to control active-active replication behavior, specifically for MySQL sinks.
• DML Event Filtering and Processing: Implemented logic to filter DML events based on active-active and soft-delete policies. This includes converting soft-delete updates to delete events when active-active is disabled, and dropping delete events when active-active is enabled.
• Active-Active Progress Tracking: Added a new ProgressTableWriter component for MySQL sinks to track replication progress for active-active tables in a dedicated system table, ensuring checkpointing and DDL handling are compatible with active-active semantics.
• Refactored MySQL Sink DML Logic: The DML generation and execution logic within the MySQL sink has been refactored into separate files (mysql_writer_dml_active_active.go, mysql_writer_dml_batch.go, mysql_writer_dml_exec.go) to improve modularity and support active-active specific UPSERT statements.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

Please refer to [Release Notes Language Style Guide](https://pingcap.github.io/tidb-dev-guide/contribute-to-tidb/release-notes-style-guide.html) to write a quality release note.

If you don't think this PR needs a release note then fill it with `None`.

Summary by CodeRabbit

• New Features

  • Toggleable active‑active replication with configurable progress and sync intervals; MySQL progress tracking and per‑connection active‑active sync metrics; new Grafana "Active Active" dashboard row.

• Enhancements

  • MySQL sink: LWW upserts for active‑active, multi‑level batching, sessionized DML execution, batched progress/DDL updates, and increased retry robustness.

• Bug Fixes

  • MySQL sink respects active‑active checkpoint handling flag.

• Tests

  • Expanded unit tests for dispatchers, writers, schema store, event filtering, and sinks.

[ti-chi-bot]

This cherry pick PR is for a release branch and has not yet been approved by triage owners.
Adding the do-not-merge/cherry-pick-not-approved label.

To merge this cherry pick:

1. It must be LGTMed and approved by the reviewers firstly.
2. For pull requests to TiDB-x branches, it must have no failed tests.
3. AFTER it has lgtm and approved labels, please wait for the cherry-pick merging approval from triage owners.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign flowbehappy for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS
• pkg/config/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

@hongyunyan This PR has conflicts, I have hold it.
Please resolve them or ask others to resolve them, then comment /unhold to remove the hold label.

[ti-chi-bot]

@ti-chi-bot: ## If you want to know how to resolve it, please read the guide in TiDB Dev Guide.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[ti-chi-bot]

@ti-chi-bot: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-error-log-review	a66e10f	link	true	/test pull-error-log-review

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances TiCDC's capabilities for active-active replication, particularly for TiDB and other MySQL-compatible downstream systems. It introduces comprehensive configuration options, refines DML event processing to handle complex filtering and transformation rules for active-active and soft-delete tables, and establishes a robust mechanism for tracking replication progress. The changes also include a modular refactoring of the MySQL sink's DML logic, improving maintainability and supporting advanced conflict resolution strategies. These updates collectively aim to provide more consistent and reliable data synchronization in active-active environments.

Highlights

• Active-Active Replication Configuration: Introduced new configuration options, enable-active-active, active-active-progress-interval, and active-active-sync-stats-interval, within the ReplicaConfig to control active-active replication behavior, specifically for MySQL/TiDB sinks.
• DML Event Filtering and Processing: Implemented logic to filter DML events based on active-active and soft-delete policies. This includes converting soft-delete updates to delete events when active-active is disabled, dropping delete events when active-active is enabled, and preventing replication loops by dropping rows with non-NULL _tidb_origin_ts in active-active mode.
• Active-Active Progress Tracking: Added a new ProgressTableWriter component for MySQL/TiDB sinks to track replication progress for active-active tables in a dedicated system table, ensuring checkpointing and DDL handling are compatible with active-active semantics.
• Refactored MySQL Sink DML Logic: The DML generation and execution logic within the MySQL sink has been refactored into separate files (mysql_writer_dml_active_active.go, mysql_writer_dml_batch.go, mysql_writer_dml_exec.go, mysql_writer_dml_session.go) to improve modularity and support active-active specific UPSERT statements and session management.
• Enhanced MySQL Sink Robustness and Metrics: Increased retry robustness for the MySQL sink, improved connection pool management, and added new Grafana metrics for tracking active-active conflict skip rows.
• Table Schema Store Refinements: The TableSchemaStore now dynamically adjusts its metadata tracking requirements based on the sink type and active-active mode, optimizing for efficiency.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• api/v2/model.go
  • Added EnableActiveActive, ActiveActiveProgressInterval, and ActiveActiveSyncStatsInterval fields to ReplicaConfig struct.
  • Updated toInternalReplicaConfigWithOriginConfig to map new active-active configuration fields.
  • Updated ToAPIReplicaConfig to include new active-active configuration fields.
• coordinator/changefeed/changefeed.go
  • Modified NeedCheckpointTsMessage to consider EnableActiveActive for MySQL sinks.
• coordinator/changefeed/changefeed_test.go
  • Added TestChangefeed_NeedCheckpointMysqlActiveActive to verify checkpoint message requirement when active-active is enabled.
  • Added TestChangefeed_NeedCheckpointMysqlDisabled to verify checkpoint message is not required when active-active is disabled for MySQL.
• downstreamadapter/dispatcher/basic_dispatcher.go
  • Imported mysql and tidbTypes packages for schema validation.
  • Added EnableActiveActive method to Dispatcher interface.
  • Added tableModeCompatibilityChecked field to BasicDispatcher to optimize schema validation.
  • Introduced AddDMLEventsToSink method to filter DML events based on active-active and soft-delete policies.
  • Updated InitializeTableSchemaStore to pass EnableActiveActive to NewTableSchemaStore.
  • Removed old AddDMLEventsToSink method.
  • Added ensureActiveActiveTableInfo and checkTableModeCompatibility methods for validating table schema requirements.
  • Added HandleError comment for clarity.
  • Integrated checkTableModeCompatibility into handleEvents for DML and DDL events.
  • Modified handleEvents to account for filtered DML events not blocking dispatcher progress.
• downstreamadapter/dispatcher/basic_dispatcher_active_active_test.go
  • Added new file for active-active specific dispatcher tests.
  • Added TestHandleEventsRejectActiveActiveTableWhenDisabled to ensure active-active tables are rejected when the mode is disabled.
  • Added TestHandleEventsRejectSoftDeleteTableWhenDisabled to ensure soft-delete tables are rejected when active-active mode is disabled.
  • Added TestHandleEventsIgnoreSpecialTableOnNonMySQLSink to verify special tables are ignored on non-MySQL sinks.
  • Added TestDDLEventsAlwaysValidateActiveActive to confirm DDL events always trigger active-active validation.
  • Added newTestBasicDispatcher helper function for dispatcher testing.
• downstreamadapter/dispatcher/basic_dispatcher_info.go
  • Added enableActiveActive field to SharedInfo struct.
  • Updated NewSharedInfo to accept enableActiveActive parameter.
  • Updated NewSharedInfo to initialize enableActiveActive.
  • Added EnableActiveActive method to BasicDispatcher.
  • Added EnableActiveActive method to SharedInfo.
• downstreamadapter/dispatcher/event_dispatcher_test.go
  • Updated newDispatcherForTest to pass enableActiveActive parameter.
  • Updated GetAllTableNames calls to include needDroppedTableName parameter.
• downstreamadapter/dispatcher/redo_dispatcher_test.go
  • Updated newRedoDispatcherForTest to pass enableActiveActive parameter.
• downstreamadapter/dispatchermanager/dispatcher_manager.go
  • Imported commonEvent package.
  • Updated NewDispatcherManager to pass EnableActiveActive to NewSharedInfo.
  • Modified checkpoint message registration logic in InitalizeTableTriggerEventDispatcher to use NeedTableNameStoreAndCheckpointTs.
• downstreamadapter/dispatchermanager/dispatcher_manager_helper.go
  • Imported commonEvent package.
  • Updated checkpoint message removal logic in removeDispatcher to use NeedTableNameStoreAndCheckpointTs.
  • Updated checkpoint message removal logic in closeAllDispatchers to use NeedTableNameStoreAndCheckpointTs.
• downstreamadapter/dispatchermanager/dispatcher_manager_test.go
  • Updated createTestDispatcher to pass enableActiveActive parameter.
  • Updated createTestManager to pass EnableActiveActive to NewSharedInfo.
• downstreamadapter/sink/kafka/sink.go
  • Updated getAllTableNames call to include needDroppedTableName parameter.
• downstreamadapter/sink/mysql/sink.go
  • Added progressTableWriter and activeActiveSyncStatsCollector fields to Sink struct.
  • Added enableActiveActive field to Sink struct.
  • Updated New function to pass new active-active parameters to NewMySQLSink.
  • Updated NewMySQLSink function signature and implementation to initialize active-active components.
  • Updated SetTableSchemaStore to also set the schema store for progressTableWriter.
  • Modified WriteBlockEvent to include logic for progressTableWriter.RemoveTables for DDL events in active-active mode.
  • Implemented AddCheckpointTs to flush active-active progress table updates.
  • Updated Close method to close activeActiveSyncStatsCollector.
• downstreamadapter/sink/mysql/sink_test.go
  • Updated getMysqlSink to pass new active-active parameters to NewMySQLSink.
  • Updated getMysqlSinkWithDDLTs to pass new active-active parameters to NewMySQLSink.
• downstreamadapter/sink/pulsar/sink.go
  • Updated getAllTableNames call to include needDroppedTableName parameter.
• go.mod
  • Updated various module dependencies, including github.com/pingcap/kvproto, github.com/pingcap/tidb, github.com/pingcap/tiflow, github.com/tikv/client-go/v2, github.com/tikv/pd, go.uber.org/zap, golang.org/x/net, and google.golang.org/grpc.
• maintainer/maintainer_controller_bootstrap.go
  • Updated getSchemaInfo and getTableInfo calls to pass enableActiveActive parameter.
  • Updated getSchemaInfo and getTableInfo function signatures to accept enableActiveActive.
• metrics/grafana/ticdc_new_arch.json
  • Added a new Grafana row for "Active Active" metrics.
  • Added a panel "Conflict Skip Rows / s" to track rows skipped due to LWW conflict resolution.
• metrics/nextgengrafana/ticdc_new_arch_next_gen.json
  • Added a new Grafana row for "Active Active" metrics.
  • Added a panel "Conflict Skip Rows / s" to track rows skipped due to LWW conflict resolution.
• metrics/nextgengrafana/ticdc_new_arch_with_keyspace_name.json
  • Added a new Grafana row for "Active Active" metrics.
  • Added a panel "Conflict Skip Rows / s" to track rows skipped due to LWW conflict resolution.
• pkg/applier/redo_test.go
  • Imported time package.
  • Updated NewMySQLSink calls to pass new active-active parameters.
• pkg/common/event/active_active.go
  • Added new file defining RowPolicyDecision enum and related constants.
  • Implemented EvaluateRowPolicy to decide how special tables should behave.
  • Implemented needConvertUpdateToDelete and isSoftDeleteTransition for soft-delete logic.
  • Implemented getSoftDeleteTimeColumnIndex and getSoftDeleteTimeColumnOffset for column validation.
  • Implemented ApplyRowPolicyDecision to mutate rows based on policy.
  • Implemented FilterDMLEvent to apply row policy decisions to DML events, including fast and slow paths.
  • Added newFilteredDMLEvent and appendRowChangeToChunk helper functions.
  • Defined rowTypeSummary and summarizeRowTypes for event analysis.
• pkg/common/event/active_active_test.go
  • Added new file for active-active event filtering tests.
  • Added tests for normal table passthrough, active-active delete skipping, soft-delete conversion, and origin TS handling.
  • Added tests for checksum clearing and error reporting on missing columns.
  • Included helper functions newTestTableInfo, newTestColumn, newDMLEventForTest, appendRowToChunk, rowSlots, and newTimestampValue.
• pkg/common/event/table_schema_store.go
  • Added NeedTableNameStoreAndCheckpointTs function to determine when table names are needed.
  • Introduced tableSchemaStoreRequirements struct and newTableSchemaStoreRequirements function for flexible metadata tracking.
  • Updated NewTableSchemaStore to accept enableActiveActive and use tableSchemaStoreRequirements.
  • Refactored AddEvent to conditionally update TableIDStore and tableNameStore based on requirements.
  • Refined initialized method to check for specific store initialization based on requirements.
  • Updated GetAllTableNames to accept needDroppedTableName parameter and apply conditional logic for dropped tables.
• pkg/common/event/table_schema_store_test.go
  • Added new file for TableSchemaStore tests.
  • Added TestTableSchemaStoreWhenMysqlSink to verify ID-based tracking for MySQL sinks.
  • Added TestTableSchemaStoreActiveActiveMetadata to verify name-based tracking for active-active MySQL sinks.
  • Added TestTableSchemaStoreWhenNonMysqlSink to verify name-based tracking for non-MySQL sinks.
  • Added TestTableSchemaStoreNonMysqlTableIDsBootstrapOnly to verify table ID handling for non-MySQL sinks.
• pkg/common/table_info.go
  • Added ActiveActiveTable and SoftDeleteTable boolean fields to TableInfo struct.
  • Added IsActiveActiveTable and IsSoftDeleteTable methods.
  • Added GetColumnInfoByName and GetColumnOffsetByName methods.
  • Updated ToTiDBTableInfo to include IsActiveActive and SoftdeleteInfo.
  • Updated newTableInfo to initialize ActiveActiveTable and SoftDeleteTable from model.TableInfo.
• pkg/config/changefeed.go
  • Added EnableActiveActive, ActiveActiveProgressInterval, and ActiveActiveSyncStatsInterval fields to ChangefeedConfig.
  • Updated ToChangefeedConfig to map new active-active configuration fields.
  • Updated VerifyAndComplete to set default values for new active-active intervals.
• pkg/config/replica_config.go
  • Added defaultActiveActiveProgressInterval and defaultActiveActiveSyncStatsInterval constants.
  • Updated defaultReplicaConfig to include EnableActiveActive and default active-active intervals.
  • Added EnableActiveActive, ActiveActiveProgressInterval, and ActiveActiveSyncStatsInterval fields to replicaConfig struct.
  • Updated ValidateAndAdjust to validate new active-active configuration options, including compatibility checks with BDR mode, sink type, and redo log.
• pkg/metrics/sink.go
  • Added ActiveActiveConflictSkipRowsCounter Prometheus counter for tracking LWW conflict skips.
  • Registered ActiveActiveConflictSkipRowsCounter in initSinkMetrics.
• pkg/pdutil/api_client.go
  • Increased defaultMaxRetry from 5 to 50.
• pkg/sink/mysql/config.go
  • Added tidbTxnModeSpecified boolean field to Config struct.
  • Added EnableActiveActive and ActiveActiveSyncStatsInterval fields to Config struct.
  • Updated New function to initialize tidbTxnModeSpecified and ActiveActiveSyncStatsInterval.
  • Modified mergeConfig to track if TiDBTxnMode was explicitly specified.
  • Updated Apply to pass tidbTxnModeSpecified to getTiDBTxnMode.
  • Updated NewMysqlConfigAndDB to set EnableActiveActive and ActiveActiveSyncStatsInterval from changefeed config.
  • Increased db.SetMaxIdleConns and db.SetMaxOpenConns by extraConn (10) for better connection pool management.
  • Updated getTiDBTxnMode to accept modeSpecified pointer.
• pkg/sink/mysql/config_test.go
  • Added testActiveActiveDefaultTiDBTxnMode to verify default TiDB transaction mode in active-active scenarios.
  • Integrated testActiveActiveDefaultTiDBTxnMode into TestGenerateDSNByConfig.
  • Updated TestApplySinkURIParamsToConfig to set tidbTxnModeSpecified to true when tidb-txn-mode is provided in URI.
• pkg/sink/mysql/helper.go
  • Modified generateDSNByConfig to set tidbTxnMode to pessimistic by default if active-active is enabled and not explicitly specified.
  • Modified GenerateDSN to validate that enable-active-active requires a TiDB downstream.
  • Modified GenerateDSN to disable tidb_translate_softdelete_sql session variable when active-active is enabled.
• pkg/sink/mysql/mysql_writer.go
  • Added dmlConnIdleTimeout constant.
  • Added cancel context function, maxDDLTsBatch, activeActiveSyncStatsCollector, activeActiveSyncStatsInterval, and dmlSession fields to Writer struct.
  • Updated NewWriter to initialize new active-active related fields and start runDMLConnLoop for DML writers.
  • Modified Flush to handle cases where no SQLs are generated (e.g., all rows filtered out).
  • Updated Close to cancel context and close dmlSession.
• pkg/sink/mysql/mysql_writer_dml.go
  • Removed DML generation and execution logic, delegating to new dedicated files.
  • Updated prepareDMLs to call genActiveActiveSQL when active-active is enabled.
  • Added genActiveActiveSQL to dispatch to active-active specific SQL generation.
• pkg/sink/mysql/mysql_writer_dml_active_active.go
  • Added new file for active-active specific DML generation logic.
  • Implemented generateActiveActiveNormalSQLs for per-row UPSERTs.
  • Implemented generateActiveActiveBatchSQLForPerEvent for per-event batching.
  • Implemented generateActiveActiveSQLForSingleEvent to merge rows from a single event.
  • Implemented generateActiveActiveBatchSQL for cross-event batching with fallback.
  • Added collectActiveActiveRows and batchSingleTxnActiveRows helper functions.
  • Introduced originTsChecker to filter rows with non-NULL _tidb_origin_ts to prevent replication loops.
• pkg/sink/mysql/mysql_writer_dml_active_active_test.go
  • Added new file for active-active DML generation tests.
  • Added TestBuildActiveActiveUpsertSQLMultiRows for multi-row UPSERT generation.
  • Added TestActiveActiveNormalSQLs for normal SQL generation in active-active mode.
  • Added TestActiveActivePerEventBatch for per-event batching in active-active mode.
  • Added TestActiveActiveCrossEventBatch for cross-event batching in active-active mode.
  • Added TestActiveActiveDropRowsWithNonNullOriginTsForTiDBDownstream to verify dropping rows with non-NULL _tidb_origin_ts.
  • Added TestActiveActiveKeepRowsWithNullOriginTsForTiDBDownstream to verify keeping rows with NULL _tidb_origin_ts.
• pkg/sink/mysql/mysql_writer_dml_batch.go
  • Added new file for DML batching logic.
  • Implemented generateNormalSQLs and generateNormalSQL for non-batched SQL generation.
  • Implemented generateSQLForSingleEvent and generateBatchSQLsPerEvent for per-event batching.
  • Implemented buildRowChangesForUnSafeBatch, generateBatchSQLInUnSafeMode, and generateBatchSQLInSafeMode for cross-event batching.
  • Implemented batchSingleTxnDmls to group and generate DMLs.
  • Implemented groupRowsByType to categorize and split rows by type.
  • Implemented genUpdateSQL for batched update statements.
• pkg/sink/mysql/mysql_writer_dml_exec.go
  • Added new file for DML execution logic.
  • Implemented execDMLWithMaxRetries for executing DMLs with retry and backoff.
  • Implemented sequenceExecute for sequential SQL execution within a transaction.
  • Implemented multiStmtExecute for multi-statement execution.
  • Implemented logDMLTxnErr for logging DML transaction errors.
• pkg/sink/mysql/mysql_writer_dml_session.go
  • Added new file defining dmlSession for managing dedicated downstream connections.
  • Implemented dmlSessionStats for tracking connection-specific statistics.
  • Implemented NewDMLSession to create a new session with idle timeout.
  • Implemented withConn to execute functions with a session-owned connection.
  • Implemented CheckStats for periodic session maintenance (stats query, idle close).
  • Implemented close, shouldCloseLocked, closeLocked, getOrCreateLocked for session management.
  • Implemented shouldCollectActiveActiveSyncStatsLocked and tryQueryActiveActiveSyncStatsLocked for stats collection.
  • Implemented runDMLConnLoop for background session maintenance.
• pkg/sink/mysql/mysql_writer_for_active_active_sync_stats.go
  • Added new file for active-active synchronization statistics.
  • Defined selectActiveActiveSyncStatsSQL and activeActiveSyncStats struct.
  • Implemented CheckActiveActiveSyncStatsSupported to verify TiDB support for stats variable.
  • Implemented ActiveActiveSyncStatsCollector to accumulate conflict statistics.
  • Implemented NewActiveActiveSyncStatsCollector, ObserveConflictSkipRows, ForgetConn, and Close methods.
  • Implemented queryActiveActiveSyncStats to query connection ID and stats from TiDB.
• pkg/sink/mysql/mysql_writer_for_ddl_ts.go
  • Imported database/sql package.
  • Updated SendDDLTsPre to use execInsertItemBatches for batch insertion of DDL TS items.
  • Updated SendDDLTs to use execInsertItemBatches for batch insertion and dropItemQueries for batch deletion of DDL TS items.
  • Added execInsertItemBatches function for batched DDL TS item insertion.
  • Renamed insertItemQuery to buildInsertItemQuery and modified to support batching.
  • Renamed dropItemQuery to dropItemQueries and modified to support batch deletion.
• pkg/sink/mysql/mysql_writer_test.go
  • Updated newTestMysqlWriter and newTestMysqlWriterForTiDB to pass nil for activeActiveSyncStatsCollector and ensure writer closure.
  • Added TestMysqlWriter_FlushNoopWhenActiveActiveRowsDropped to verify no-op flush when active-active rows are filtered out.
  • Updated TestMysqlWriter_FlushSyncPointEvent to pass false for enableActiveActive to NewTableSchemaStore.
• pkg/sink/mysql/progress_table_writer.go
  • Added new file for active-active progress table writer.
  • Defined progressTableName and maxBatchSize constants.
  • Implemented ProgressTableWriter struct with context, DB, changefeed ID, schema store, and intervals.
  • Implemented NewProgressTableWriter to create a new instance.
  • Implemented SetTableSchemaStore to inject the schema store.
  • Implemented Flush to write checkpoint information for tracked tables, including throttling and DDL commit TS checks.
  • Implemented flushBatch to write batches of progress rows.
  • Implemented initProgressTable to lazily create the system database and progress table.
  • Implemented RemoveTables to remove progress rows for dropped tables or databases.
  • Implemented removeTableBatch and removeDatabase for batch deletion of progress rows.
• pkg/sink/mysql/progress_table_writer_test.go
  • Added new file for ProgressTableWriter tests.
  • Added newTestTableSchemaStore helper function.
  • Added TestProgressTableWriterFlushSingleBatch to verify single-batch flushing.
  • Added TestProgressTableWriterFlushMultiBatch to verify multi-batch flushing.
  • Included helper functions setTestClusterID, expectProgressTableInit, and expectProgressInsert.
• pkg/sink/mysql/sql_builder.go
  • Imported zap package.
  • Added buildActiveActiveUpsertSQL function to construct UPSERT statements for active-active replication with last-write-wins logic.
• pkg/upstream/upstream.go
  • Increased defaultMaxRetry from 5 to 50.
• tests/integration_tests/run_light_it_in_ci.sh
  • Removed ddl_default_current_timestamp from the G03 test group.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[gemini-code-assist]

Code Review

This pull request introduces foundational support for active-active replication, including new configuration options, DML event filtering, and progress tracking for MySQL-compatible sinks. The changes are extensive and well-structured, with significant refactoring of the MySQL sink logic for better modularity.

However, there are several critical issues that must be addressed:

• Unresolved merge conflicts are present in go.mod, pkg/common/table_info.go, and pkg/sink/mysql/mysql_writer_dml.go.
• The default maximum retry count has been increased tenfold in two places, which could mask underlying instability and needs justification.

Additionally, there is a minor opportunity for code simplification by removing a redundant check in the dispatcher logic.

Overall, the PR is a significant step towards active-active support, but the merge conflicts must be resolved before it can be considered for merging.

[gemini-code-assist]

This file contains unresolved merge conflict markers (<<<<<<<, =======, >>>>>>>). These must be resolved before the pull request can be merged. Please clean up the go.mod file to ensure it's in a valid state.

[gemini-code-assist]

This file contains unresolved merge conflict markers (<<<<<<<, =======, >>>>>>>). These must be resolved before the pull request can be merged. Please clean up the go.mod file to ensure it's in a valid state.

[gemini-code-assist]

This file contains unresolved merge conflict markers (<<<<<<<, =======, >>>>>>>). These must be resolved before the pull request can be merged. Please clean up the go.mod file to ensure it's in a valid state.

[gemini-code-assist]

The defaultMaxRetry has been increased from 5 to 50. This is a tenfold increase and could lead to very long retry cycles (up to several minutes with backoff), potentially masking underlying instability issues in PD or the network. Could you provide some context for this change? If this is to handle specific known failure scenarios, it would be good to add a comment explaining the reasoning.

[gemini-code-assist]

Similar to the change in pdutil, defaultMaxRetry is increased from 5 to 50 here. This large increase in retries might hide persistent issues with the upstream TiKV. Please add a comment explaining the rationale behind this change, or consider if a more moderate value would be sufficient.

[gemini-code-assist]

This check for the column offset is redundant. The subsequent call to tableInfo.GetColumnInfoByName on line 382 also verifies the column's existence. If that call succeeds, the offset is guaranteed to exist. You can safely remove this block.