-
Notifications
You must be signed in to change notification settings - Fork 5.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
tidb panic after inject one of AZ network partition with pitr running #52597
Comments
/assign YuJuncen |
@Lily2025: The label(s) In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository. |
Could you upload the logs from |
Bug Report
Please answer these questions before submitting your issue. Thanks!
1. Minimal reproduce step (Required)
1、run pitr
2、run tpcc with 2w warehouse and 32 threads
3、inject network partition between one of az and other az last for 10mins
fault start time:2024-04-12 23:16:25
fault end time:2024-04-12 23:26:32
2. What did you expect to see? (Required)
no panic
3. What did you see instead (Required)
tidb panic after fault recover
2024-04-12 23:27:44 log="/tidb-server --store=tikv --advertise-address=tc-tidb-1.tc-tidb-peer.endless-ha-test-oltp-pitr-tps-7504383-1-172.svc --host=0.0.0.0 --path=tc-pd:2379 --config=/etc/tidb/tidb.toml\n" 2024-04-12 23:27:44 log="start tidb-server ...\n" 2024-04-12 23:27:33 log="\t/workspace/source/tidb/br/pkg/streamhelper/flush_subscriber.go:150 +0x78\n" 2024-04-12 23:27:33 log="created by github.com/pingcap/tidb/br/pkg/streamhelper.spawnJoinable in goroutine 16842\n" 2024-04-12 23:27:33 log="\t/workspace/source/tidb/br/pkg/streamhelper/flush_subscriber.go:152 +0x4d\n" 2024-04-12 23:27:33 log="github.com/pingcap/tidb/br/pkg/streamhelper.spawnJoinable.func1()\n" 2024-04-12 23:27:33 log="\t/workspace/source/tidb/br/pkg/streamhelper/flush_subscriber.go:237 +0x27\n" 2024-04-12 23:27:33 log="github.com/pingcap/tidb/br/pkg/streamhelper.(*subscription).doConnect.func1()\n" 2024-04-12 23:27:33 log="\t/workspace/source/tidb/br/pkg/streamhelper/flush_subscriber.go:280 +0xac5\n" 2024-04-12 23:27:33 log="github.com/pingcap/tidb/br/pkg/streamhelper.(*subscription).listenOver(0xc0b24bf600, {0x66287d8, 0xc086a1e420}, {0x663d180, 0xc057c9afc0})\n" 2024-04-12 23:27:33 log="goroutine 1090341181 [running]:\n" 2024-04-12 23:27:33 log="\n" 2024-04-12 23:27:33 log="panic: send on closed channel\n"
4. What is your TiDB version? (Required)
./tidb-server -V
Release Version: v8.1.0
Edition: Community
Git Commit Hash: 6bf6755
Git Branch: HEAD
UTC Build Time: 2024-04-12 11:10:02
GoVersion: go1.21.4
Race Enabled: false
Check Table Before Drop: false
Store: unistore
2024-04-12T23:06:17.157+0800
The text was updated successfully, but these errors were encountered: