Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ticdc crash when inject pdleader network_partition #8562

Closed
Lily2025 opened this issue Mar 17, 2023 · 3 comments · Fixed by #8602
Closed

ticdc crash when inject pdleader network_partition #8562

Lily2025 opened this issue Mar 17, 2023 · 3 comments · Fixed by #8602

Comments

@Lily2025
Copy link

Lily2025 commented Mar 17, 2023

What did you do?

1、create changefeed
2、run workload tpcc with 1000 warehouse and 32 thread
3、inject pdleader network_partition last for 10m

What did you expect to see?

ticdc should not crash

What did you see instead?

ticdc crash

image

[2023/03/08 15:24:37.743 +08:00] [ERROR] [capture.go:285] ["reset capture failed"] [error="[CDC:ErrNewCaptureFailed]new capture failed: [CDC:ErrNewCaptureFailed]new capture failed: context deadline exceeded"] [errorVerbose="[CDC:ErrNewCaptureFailed]new capture failed: [CDC:ErrNewCaptureFailed]new capture failed: context deadline exceeded\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\tgithub.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/normalize.go:164\ngithub.com/pingcap/tiflow/pkg/errors.WrapError\n\tgithub.com/pingcap/tiflow/pkg/errors/helper.go:34\ngithub.com/pingcap/tiflow/cdc/capture.(*captureImpl).reset\n\tgithub.com/pingcap/tiflow/cdc/capture/capture.go:194\ngithub.com/pingcap/tiflow/cdc/capture.(*captureImpl).Run\n\tgithub.com/pingcap/tiflow/cdc/capture/capture.go:283\ngithub.com/pingcap/tiflow/cdc/server.(*server).run.func1\n\tgithub.com/pingcap/tiflow/cdc/server/server.go:315\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\nruntime.goexit\n\truntime/asm_amd64.s:1598"]

Versions of the cluster

./cdc version
Release Version: v6.7.0-alpha
Git Commit Hash: f43dfb2
Git Branch: heads/refs/tags/v6.7.0-alpha
UTC Build Time: 2023-03-07 11:34:11
Go Version: go version go1.20.1 linux/amd64
Failpoint Build: false

current status of DM cluster (execute query-status <task-name> in dmctl)

(paste current status of DM cluster here)
@Lily2025 Lily2025 added area/dm Issues or PRs related to DM. type/bug This is a bug. labels Mar 17, 2023
@github-actions github-actions bot added this to Need Triage in Question and Bug Reports Mar 17, 2023
@Lily2025
Copy link
Author

/remove-area dm
/area ticdc

@ti-chi-bot ti-chi-bot added area/ticdc Issues or PRs related to TiCDC. and removed area/dm Issues or PRs related to DM. labels Mar 17, 2023
@fubinzh
Copy link

fubinzh commented Mar 17, 2023

/severity critical

@nongfushanquan
Copy link
Contributor

/assign @overvenus

Question and Bug Reports automation moved this from Need Triage to Done Mar 22, 2023
overvenus added a commit to ti-chi-bot/tiflow that referenced this issue Mar 24, 2023
3AceShowHand pushed a commit to 3AceShowHand/tiflow that referenced this issue Apr 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment