kafka: make the verify lightweight (#5617) by ti-chi-bot · Pull Request #5811 · pingcap/ticdc

ti-chi-bot · 2026-07-29T10:16:46Z

This PR backports #5617 to release-8.5.

What problem does this PR solve?

Issue Number: close #5618, ref #5719, ref #5720, ref #5790

This PR brings a related set of Kafka fixes from master to release-8.5 in one backport:

Kafka sink verification should stay lightweight and avoid constructing runtime-only components.
Claim-check storage should be shared by all encoders owned by one sink.
replication-factor should be validated when TiCDC needs to create a topic, not for an existing topic.
Kafka sink errors should use the unified TiCDC pkg/errors definitions and preserve the same error behavior as master.

What is changed and how it works?

#5617 / #5811: lightweight Kafka verification

Runs the shared protocol, topic, sink-option, encoder, router, and column-selector validation directly.
Uses Kafka admin metadata and validate-only topic creation without creating a topic during verification.
Leaves real topic creation to normal sink startup.
Removes the test-only Kafka factory creator indirection.

Additional cherry-picks carried by this PR

kafka: share one claimcheck instance across encoders (#5718) #5812: shares one claim-check instance across the Kafka sink's encoders and closes it with the sink components.
kafka: verify replication-factor when need to create the topic (#5715) #5813: validates replication-factor against broker min.insync.replicas only when a topic needs to be created.
kafka: unify sink errors and replace failpoint tests (#5786) #5815: unifies Kafka sink errors under TiCDC pkg/errors and updates the related error-path tests.

Check List

Tests

Unit test

Commands run locally:

go test --tags=intest ./pkg/sink/kafka ./downstreamadapter/sink/kafka ./downstreamadapter/sink/topicmanager -count=1
go test --run '^$' --tags=intest ./...
bash -n tests/integration_tests/kafka_big_messages/run.sh
git diff --check

Questions

Will it cause performance regression or break compatibility?

No performance regression is expected. Lightweight verification avoids startup-only work, and sharing one claim-check instance reduces duplicated resources. Kafka error classification is aligned with master; no user-facing Kafka configuration or protocol format is changed by this PR.

Do you need to update user documentation, design documentation or monitoring documentation?

No. These are internal Kafka sink verification, lifecycle, validation, and error-handling changes.

Release note

None

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot · 2026-07-29T10:16:50Z

@3AceShowHand This PR has conflicts, I have hold it.
Please resolve them or ask others to resolve them, then comment /unhold to remove the hold label.

ti-chi-bot · 2026-07-29T10:16:53Z

@ti-chi-bot: ## If you want to know how to resolve it, please read the guide in TiDB Dev Guide.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

…5617-to-release-8.5 # Conflicts: # downstreamadapter/sink/kafka/helper.go

ti-chi-bot · 2026-07-30T08:18:56Z

Cherry-pick conflicts appear resolved; removing the do-not-merge/hold label.

coderabbitai · 2026-07-30T08:18:58Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: ac8de7ca-80d8-4a45-a80b-8bc70e0b2885

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

…-to-release-8.5 # Conflicts: # downstreamadapter/sink/kafka/helper.go # downstreamadapter/sink/kafka/sink.go # downstreamadapter/sink/kafka/sink_test.go

…-to-release-8.5 # Conflicts: # downstreamadapter/sink/kafka/sink.go # downstreamadapter/sink/kafka/sink_test.go # downstreamadapter/sink/topicmanager/kafka_topic_manager_test.go # pkg/sink/kafka/options_test.go

…-to-release-8.5 # Conflicts: # downstreamadapter/sink/kafka/helper.go # downstreamadapter/sink/kafka/sink.go # downstreamadapter/sink/kafka/sink_test.go # downstreamadapter/sink/topicmanager/kafka_topic_manager.go # downstreamadapter/sink/topicmanager/kafka_topic_manager_test.go # pkg/sink/kafka/admin_test.go # pkg/sink/kafka/claimcheck/claim_check_test.go # pkg/sink/kafka/cluster_admin_client_mock.go # pkg/sink/kafka/options.go # pkg/sink/kafka/options_test.go # pkg/sink/kafka/sarama_sync_producer_test.go

3AceShowHand · 2026-07-30T09:00:49Z

/test all

3AceShowHand · 2026-07-30T09:37:39Z

/test all

3AceShowHand · 2026-07-30T10:06:28Z

/retest

3AceShowHand · 2026-07-30T10:31:12Z

/test all

ti-chi-bot · 2026-07-30T10:55:59Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: 3AceShowHand, nongfushanquan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [3AceShowHand,nongfushanquan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

3AceShowHand · 2026-07-30T10:57:54Z

/retest

3AceShowHand · 2026-07-30T11:33:37Z

/retest

This is an automated cherry-pick of pingcap#5617

84e3900

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot Bot added the do-not-merge/cherry-pick-not-approved label Jul 29, 2026

ti-chi-bot mentioned this pull request Jul 29, 2026

kafka: make the verify lightweight #5617

Merged

ti-chi-bot assigned 3AceShowHand Jul 29, 2026

3AceShowHand added 3 commits July 29, 2026 10:22

This is an automated cherry-pick of pingcap#5718

45dfe16

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

This is an automated cherry-pick of pingcap#5715

f7a1a67

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

This is an automated cherry-pick of pingcap#5786

28605d0

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot Bot added cherry-pick-approved Cherry pick PR approved by release team. and removed do-not-merge/cherry-pick-not-approved labels Jul 30, 2026

Merge remote-tracking branch 'upstream/release-8.5' into cherry-pick-…

efe799b

…5617-to-release-8.5 # Conflicts: # downstreamadapter/sink/kafka/helper.go

ti-chi-bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 30, 2026

3AceShowHand added 3 commits July 30, 2026 16:27

Merge remote-tracking branch 'upstream/pr-5812' into cherry-pick-5617…

1fb9c7a

…-to-release-8.5 # Conflicts: # downstreamadapter/sink/kafka/helper.go # downstreamadapter/sink/kafka/sink.go # downstreamadapter/sink/kafka/sink_test.go

Merge remote-tracking branch 'upstream/pr-5813' into cherry-pick-5617…

8be9f87

…-to-release-8.5 # Conflicts: # downstreamadapter/sink/kafka/sink.go # downstreamadapter/sink/kafka/sink_test.go # downstreamadapter/sink/topicmanager/kafka_topic_manager_test.go # pkg/sink/kafka/options_test.go

ti-chi-bot Bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jul 30, 2026

This was referenced Jul 30, 2026

kafka: unify sink errors and replace failpoint tests (#5786) #5815

Closed

kafka: verify replication-factor when need to create the topic (#5715) #5813

Closed

kafka: share one claimcheck instance across encoders (#5718) #5812

Closed

kafka: defer max message bytes backport

fb9a32f

3AceShowHand approved these changes Jul 30, 2026

View reviewed changes

ti-chi-bot Bot added the approved label Jul 30, 2026

fix consumer

f0a2830

nongfushanquan approved these changes Jul 30, 2026

View reviewed changes

ti-chi-bot Bot merged commit 7aced84 into pingcap:release-8.5 Jul 30, 2026
17 checks passed

Navigation Menu

Uh oh!

kafka: make the verify lightweight (#5617) - #5811

kafka: make the verify lightweight (#5617)#5811
ti-chi-bot[bot] merged 10 commits into
pingcap:release-8.5from
ti-chi-bot:cherry-pick-5617-to-release-8.5

ti-chi-bot commented Jul 29, 2026 •

edited by 3AceShowHand

Loading

Uh oh!

ti-chi-bot commented Jul 29, 2026

Uh oh!

ti-chi-bot Bot commented Jul 29, 2026

Uh oh!

ti-chi-bot commented Jul 30, 2026

Uh oh!

coderabbitai Bot commented Jul 30, 2026 •

edited

Loading

Review skipped

Uh oh!

3AceShowHand commented Jul 30, 2026

Uh oh!

3AceShowHand commented Jul 30, 2026

Uh oh!

3AceShowHand commented Jul 30, 2026

Uh oh!

3AceShowHand commented Jul 30, 2026

Uh oh!

ti-chi-bot Bot commented Jul 30, 2026

Uh oh!

3AceShowHand commented Jul 30, 2026

Uh oh!

3AceShowHand commented Jul 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

ti-chi-bot commented Jul 29, 2026 • edited by 3AceShowHand Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What is changed and how it works?

#5617 / #5811: lightweight Kafka verification

Additional cherry-picks carried by this PR

Check List

Tests

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

Uh oh!

ti-chi-bot commented Jul 29, 2026

Uh oh!

ti-chi-bot Bot commented Jul 29, 2026

Uh oh!

ti-chi-bot commented Jul 30, 2026

Uh oh!

coderabbitai Bot commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

3AceShowHand commented Jul 30, 2026

Uh oh!

3AceShowHand commented Jul 30, 2026

Uh oh!

3AceShowHand commented Jul 30, 2026

Uh oh!

3AceShowHand commented Jul 30, 2026

Uh oh!

ti-chi-bot Bot commented Jul 30, 2026

Uh oh!

3AceShowHand commented Jul 30, 2026

Uh oh!

3AceShowHand commented Jul 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ti-chi-bot commented Jul 29, 2026 •

edited by 3AceShowHand

Loading

coderabbitai Bot commented Jul 30, 2026 •

edited

Loading