Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Invalid Ticdc parameter partition-num for Tidb7.1.1 #9952

Closed
cloudWang001 opened this issue Oct 24, 2023 · 3 comments · Fixed by #9955
Closed

Invalid Ticdc parameter partition-num for Tidb7.1.1 #9952

cloudWang001 opened this issue Oct 24, 2023 · 3 comments · Fixed by #9955
Labels
affects-6.5 affects-7.1 affects-7.2 affects-7.3 affects-7.4 affects-7.5 area/ticdc Issues or PRs related to TiCDC. found/community The community has encountered this bug. severity/major This is a major bug. type/bug This is a bug.

Comments

@cloudWang001
Copy link

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

  1. Create a Kafka topic with 3 partitions
  2. Create Ticdc changefeed:
    curl -X POST http://xxx.xxx.xxx.xxx:8300/api/v2/changefeeds -d ‘{“changefeed_id”:“test-patition”,“start_ts”: 445150430914150401,“sink_uri”:“kafka://XXX.XXX.XXX.XXX:9092/kafka-task1?enable-tidb-extension=true&partition-num=1&replication-factor=1&max-message-bytes=67108864”,“replica_config”: {“filter”: {“rules”:[“test_cdc."]},“ignore_ineligible_table”: true,“mounter”: {“worker_num”: 16},“sink”: {“dispatchers”: [{“matcher”: ["test_cdc.”],“partition”: “index-value”}],“protocol”: “canal-json”}}}’

2. What did you expect to see? (Required)

Kafka only has one partition with synchronized data, and the other two partitions should not have synchronized data.

3. What did you see instead (Required)

All three partitions of Kafka have synchronized data.

4. What is your TiDB version? (Required)

Release Version: v7.1.1
Edition: Community
Git Commit Hash: cf441574864be63938524e7dfcf7cc659edc3dd8
Git Branch: heads/refs/tags/v7.1.1
UTC Build Time: 2023-07-19 10:20:53
GoVersion: go1.20.6
Race Enabled: false
TiKV Min Version: 6.2.0-alpha
Check Table Before Drop: false
Store: tikv

5. PS

1.The parameter replication-factor=1 or 3 does not affect the results.

2.The results obtained by using API v2 or the command line cdc cli changefeed create are the same.

3.The parameter partition-num is valid in Tidb4.0.15, but it becomes invalid in Tidb7.1.1:
Tidb7.1.1:
(1) partition-num=3, topic partitions=3 : 3 partitions have synchronized data
(2) partition-num=2, topic partitions=3 : 3 partitions have synchronized data
(3) partition-num=1, topic partitions=3 : 3 partitions have synchronized data

(4) partition-num=2, topic partitions=2 : 2 partitions have synchronized data
(5) partition-num=1, topic partitions=2 : 2 partitions have synchronized data

Tidb4.0.15:
(1) partition-num=3, topic partitions=3 : 3 partitions have synchronized data
(2) partition-num=2, topic partitions=3 : 2 partitions have synchronized data
(3) partition-num=1, topic partitions=3 : 1 partitions have synchronized data

(4) partition-num=2, topic partitions=2 : 2 partitions have synchronized data
(5) partition-num=1, topic partitions=2 : 1 partitions have synchronized data

@cloudWang001 cloudWang001 added the type/bug This is a bug. label Oct 24, 2023
@fubinzh
Copy link

fubinzh commented Oct 25, 2023

/severity major

@fubinzh
Copy link

fubinzh commented Oct 25, 2023

CDC had below WARN log printed, however all partitions have messages dispatched.

["number of partition specified in sink-uri is less than that of the actual topic. Some partitions will not have messages dispatched to"] [sinkUriPartitions=1] [topicPartitions=3]

Question and Bug Reports automation moved this from In Progress to Done Oct 30, 2023
ti-chi-bot bot pushed a commit that referenced this issue Oct 30, 2023
ti-chi-bot bot pushed a commit that referenced this issue Oct 31, 2023
ti-chi-bot bot pushed a commit that referenced this issue Nov 1, 2023
ti-chi-bot bot pushed a commit that referenced this issue Nov 10, 2023
@seiya-annie
Copy link

/found community

@ti-chi-bot ti-chi-bot bot added the found/community The community has encountered this bug. label Jun 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment