tidb connection panic after fault recover from one of tidb network partition #7926
Labels
affects-7.5
severity/critical
The issue's severity is critical.
type/bug
The issue is confirmed as a bug.
Projects
What did you do?
1、run tpcc
2、inject one of tidb network partition last for 10mins and recover
What did you expect to see?
no panic
What did you see instead?
tidb connection panic after fault recover
[2024/03/16 12:02:50.582 +08:00] [ERROR] [conn.go:1150] ["connection running loop panic"] [keyspaceName=keyspace_a] [keyspaceID=1] [conn=4069787314631476933] [gw_conn=] [lastSQL="update district set d_ytd = d_ytd + ? where d_w_id = ? and d_id = ?"] [err="inconsistent label cardinality: expected 2 label values but got 1 in []string{"default"}"] [stack="github.com/pingcap/tidb/server.(*clientConn).Run.func1\n\t/go/src/github.com/pingcap/tidb/server/conn.go:1153\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:914\ngithub.com/pingcap/tidb/executor.(*ExecStmt).Exec.func1\n\t/go/src/github.com/pingcap/tidb/executor/adapter.go:463\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:914\ngithub.com/prometheus/client_golang/prometheus.(*CounterVec).WithLabelValues\n\t/go/cache/github.com/prometheus/client_golang@v1.18.0/prometheus/counter.go:284\ngithub.com/tikv/pd/client/resource_group/controller.(*ResourceGroupsController).IsBackgroundRequest\n\t/go/cache/github.com/tikv/pd/client@v0.0.0-20240314040809-c2c5d849cef5/resource_group/controller/controller.go:567\ngithub.com/tikv/client-go/v2/internal/client.buildResourceControlInterceptor.func1.1\n\t/go/cache/github.com/tikv/client-go/v2@v2.0.8-0.20240314054243-d5fcee418f95/internal/client/client_interceptor.go:104\ngithub.com/tikv/client-go/v2/internal/client.interceptedClient.SendRequest\n\t/go/cache/github.com/tikv/client-go/v2@v2.0.8-0.20240314054243-d5fcee418f95/internal/client/client_interceptor.go:58\ngithub.com/tikv/client-go/v2/internal/client.reqCollapse.SendRequest\n\t/go/cache/github.com/tikv/client-go/v2@v2.0.8-0.20240314054243-d5fcee418f95/internal/client/client_collapse.go:74\ngithub.com/tikv/client-go/v2/internal/locate.(*RegionRequestSender).sendReqToRegion\n\t/go/cache/github.com/tikv/client-go/v2@v2.0.8-0.20240314054243-d5fcee418f95/internal/locate/region_request.go:1712\ngithub.com/tikv/client-go/v2/internal/locate.(*RegionRequestSender).SendReqCtx\n\t/go/cache/github.com/tikv/client-go/v2@v2.0.8-0.20240314054243-d5fcee418f95/internal/locate/region_request.go:1470\ngithub.com/tikv/client-go/v2/internal/locate.(*RegionRequestSender).SendReq\n\t/go/cache/github.com/tikv/client-go/v2@v2.0.8-0.20240314054243-d5fcee418f95/internal/locate/region_request.go:252\ngithub.com/tikv/client-go/v2/txnkv/transaction.actionPessimisticLock.handleSingleBatch\n\t/go/cache/github.com/tikv/client-go/v2@v2.0.8-0.20240314054243-d5fcee418f95/txnkv/transaction/pessimistic.go:189\ngithub.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnBatches\n\t/go/cache/github.com/tikv/client-go/v2@v2.0.8-0.20240314054243-d5fcee418f95/txnkv/transaction/2pc.go:1084\ngithub.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnGroupMutations\n\t/go/cache/github.com/tikv/client-go/v2@v2.0.8-0.20240314054243-d5fcee418f95/txnkv/transaction/2pc.go:989\ngithub.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnMutations\n\t/go/cache/github.com/tikv/client-go/v2@v2.0.8-0.20240314054243-d5fcee418f95/txnkv/transaction/2pc.go:808\ngithub.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).pessimisticLockMutations\n\t/go/cache/github.com/tikv/client-go/v2@v2.0.8-0.20240314054243-d5fcee418f95/txnkv/transaction/pessimistic.go:601\ngithub.com/tikv/client-go/v2/txnkv/transaction.(*KVTxn).lockKeys\n\t/go/cache/github.com/tikv/client-go/v2@v2.0.8-0.20240314054243-d5fcee418f95/txnkv/transaction/txn.go:1107\ngithub.com/tikv/client-go/v2/txnkv/transaction.(*KVTxn).LockKeysFunc\n\t/go/cache/github.com/tikv/client-go/v2@v2.0.8-0.20240314054243-d5fcee418f95/txnkv/transaction/txn.go:919\ngithub.com/pingcap/tidb/store/driver/txn.(*tikvTxn).LockKeysFunc\n\t/go/src/github.com/pingcap/tidb/store/driver/txn/txn_driver.go:91\ngithub.com/pingcap/tidb/session.(*LazyTxn).LockKeysFunc\n\t/go/src/github.com/pingcap/tidb/session/txn.go:477\ngithub.com/pingcap/tidb/session.(*LazyTxn).LockKeys\n\t/go/src/github.com/pingcap/tidb/session/txn.go:452\ngithub.com/pingcap/tidb/executor.doLockKeys\n\t/go/src/github.com/pingcap/tidb/executor/executor.go:1369\ngithub.com/pingcap/tidb/executor.(*PointGetExecutor).lockKeyBase\n\t/go/src/github.com/pingcap/tidb/executor/point_get.go:409\ngithub.com/pingcap/tidb/executor.(*PointGetExecutor).lockKeyIfNeeded\n\t/go/src/github.com/pingcap/tidb/executor/point_get.go:384\ngithub.com/pingcap/tidb/executor.(*PointGetExecutor).Next\n\t/go/src/github.com/pingcap/tidb/executor/point_get.go:256\ngithub.com/pingcap/tidb/executor.Next\n\t/go/src/github.com/pingcap/tidb/executor/executor.go:327\ngithub.com/pingcap/tidb/executor.(*UpdateExec).updateRows\n\t/go/src/github.com/pingcap/tidb/executor/update.go:265\ngithub.com/pingcap/tidb/executor.(*UpdateExec).Next\n\t/go/src/github.com/pingcap/tidb/executor/update.go:239\ngithub.com/pingcap/tidb/executor.Next\n\t/go/src/github.com/pingcap/tidb/executor/executor.go:327\ngithub.com/pingcap/tidb/executor.(*ExecStmt).next\n\t/go/src/github.com/pingcap/tidb/executor/adapter.go:1202\ngithub.com/pingcap/tidb/executor.(*ExecStmt).handleNoDelayExecutor\n\t/go/src/github.com/pingcap/tidb/executor/adapter.go:947\ngithub.com/pingcap/tidb/executor.(*ExecStmt).handlePessimisticDML\n\t/go/src/github.com/pingcap/tidb/executor/adapter.go:1008\ngithub.com/pingcap/tidb/executor.(*ExecStmt).handleNoDelay\n\t/go/src/github.com/pingcap/tidb/executor/adapter.go:770\ngithub.com/pingcap/tidb/executor.(*ExecStmt).Exec\n\t/go/src/github.com/pingcap/tidb/executor/adapter.go:568\ngithub.com/pingcap/tidb/session.runStmt\n\t/go/src/github.com/pingcap/tidb/session/session.go:2413\ngithub.com/pingcap/tidb/session.(*session).ExecuteStmt\n\t/go/src/github.com/pingcap/tidb/session/session.go:2262\ngithub.com/pingcap/tidb/server.(*TiDBContext).ExecuteStmt\n\t/go/src/github.com/pingcap/tidb/server/driver_tidb.go:294\ngithub.com/pingcap/tidb/server.(*clientConn).executePreparedStmtAndWriteResult\n\t/go/src/github.com/pingcap/tidb/server/conn_stmt.go:308\ngithub.com/pingcap/tidb/server.(*clientConn).executePlanCacheStmt\n\t/go/src/github.com/pingcap/tidb/server/conn_stmt.go:236\ngithub.com/pingcap/tidb/server.(*clientConn).handleStmtExecute\n\t/go/src/github.com/pingcap/tidb/server/conn_stmt.go:227\ngithub.com/pingcap/tidb/server.(*clientConn).dispatch\n\t/go/src/github.com/pingcap/tidb/server/conn.go:1483\ngithub.com/pingcap/tidb/server.(*clientConn).Run\n\t/go/src/github.com/pingcap/tidb/server/conn.go:1234\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/go/src/github.com/pingcap/tidb/server/server.go:755"]
[2024/03/16 12:02:50.582 +08:00] [INFO] [server.go:733] ["connection closed"] [keyspaceName=keyspace_a] [keyspaceID=1] [conn=4069787314631476933] [gw_conn=] [remoteAddr=10.233.103.73:50984] [connectionID=4069787314631476933]
What version of PD are you using (pd-server -V)?
./pd-server -V
Release Version: v7.1.1-serverless
Edition: Community
Git Commit Hash: https://github.com/tidbcloud/pd-cse/commit/a502745f1d93745e03cab261619d1c3d3c8d4e22
Git Branch: release-7.1-keyspace
UTC Build Time: 2024-03-12 07:25:44
2024-03-16T11:42:40.444+0800 INFO k8s/client.go:135
The text was updated successfully, but these errors were encountered: