Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

resolved ts and checkpoint ts not advance when kv-client.enable-multiplexing = true #9673

Closed
fubinzh opened this issue Sep 5, 2023 · 7 comments · Fixed by #9963
Closed

resolved ts and checkpoint ts not advance when kv-client.enable-multiplexing = true #9673

fubinzh opened this issue Sep 5, 2023 · 7 comments · Fixed by #9963
Labels
affects-7.4 affects-7.5 area/ticdc Issues or PRs related to TiCDC. found/automation Bugs found by automation cases severity/critical This is a critical bug. type/bug This is a bug.
Projects
Question and Bug Reports

Comments

@fubinzh
Copy link

fubinzh commented Sep 5, 2023

What did you do?

  1. create mysql changefeed
  2. pause changefeed
  3. run tpcc workload
  4. resume changefeed

What did you expect to see?

  1. resolved ts and changefeed checkpoint should advance normally.

What did you see instead?

  1. resolved ts and changefeed checkpoint stucks. (this issue is not stably reproduciable)
    image
    image

Versions of the cluster

CDC version:
["Welcome to Change Data Capture (CDC)"] [release-version=v7.4.0-alpha] [git-hash=74673c3cad726d87e2f7901a3b80f77f4f26e991] [git-branch=heads/refs/tags/v7.4.0-alpha]
@fubinzh fubinzh added area/ticdc Issues or PRs related to TiCDC. type/bug This is a bug. labels Sep 5, 2023
@github-actions github-actions bot added this to Need Triage in Question and Bug Reports Sep 5, 2023
@fubinzh
Copy link
Author

fubinzh commented Sep 5, 2023

ticdc-0_2220647.log
ticdc-1_2220647.log
ticdc-2_2220647.log

@fubinzh
Copy link
Author

fubinzh commented Sep 5, 2023

/severity moderate
/found automation

@ti-chi-bot ti-chi-bot bot added severity/moderate This is a moderate bug. found/automation Bugs found by automation cases labels Sep 5, 2023
@fubinzh
Copy link
Author

fubinzh commented Sep 28, 2023

This issue still seen in latest master.

@fubinzh
Copy link
Author

fubinzh commented Sep 28, 2023

/remove-severity moderate
/severity major

@fubinzh
Copy link
Author

fubinzh commented Oct 7, 2023

This issue seen in latest master in several scenarios.

/remove-severity major
/severity critical

@ti-chi-bot ti-chi-bot bot added severity/critical This is a critical bug. and removed severity/major This is a major bug. labels Oct 7, 2023
@hicqu hicqu mentioned this issue Oct 9, 2023
Merged
ti-chi-bot bot pushed a commit that referenced this issue Oct 9, 2023
@hicqu
cdc: disable kv-client.enable-multiplexing by default (#9844)
f55de7f 
close #9673
@fubinzh fubinzh changed the title resolved ts and checkpoint ts not advance resolved ts and checkpoint ts not advance when kv-client.enable-multiplexing = true Oct 11, 2023
@flowbehappy
Copy link
Collaborator

@fubinzh Hi, I wonder is there any workaround for this issue?

@fubinzh
Copy link
Author

fubinzh commented Oct 23, 2023

@flowbehappy This issue only happens on master, and you should be able to workaround this issue with cdc server configuration kv-client.enable-multiplexing = false.

@hicqu hicqu mentioned this issue Oct 30, 2023
Merged
Question and Bug Reports automation moved this from Need Triage to Done Oct 31, 2023
ti-chi-bot bot pushed a commit that referenced this issue Oct 31, 2023
@hicqu
kv-client(cdc): fix the bug that region errors can be lost (#9963)
44a7474 
close #9673
@ti-chi-bot ti-chi-bot mentioned this issue Oct 31, 2023
Merged
ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this issue Oct 31, 2023
@hicqu @ti-chi-bot
kv-client(cdc): fix the bug that region errors can be lost (pingcap#9963
17f2dfd 
)

close pingcap#9673
ti-chi-bot bot pushed a commit that referenced this issue Oct 31, 2023
@ti-chi-bot
kv-client(cdc): fix the bug that region errors can be lost (#9963) (#…
453aea8 
…9977)

close #9673
@nongfushanquan nongfushanquan mentioned this issue Nov 14, 2023
Merged
18 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
affects-7.4 affects-7.5 area/ticdc Issues or PRs related to TiCDC. found/automation Bugs found by automation cases severity/critical This is a critical bug. type/bug This is a bug.
Projects
Question and Bug Reports
Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

kv-client(cdc): fix the bug that region errors can be lost hicqu/ticdc
5 participants
@flowbehappy @jebter @fubinzh @hicqu @ti-chi-bot