Hit connection running loop panic in tidb when do patrol check in serverless prod env

[mayjiang0203]

Bug Report

What did you do?

What did you expect to see?

What did you see instead?

2023/05/17 13:58:08.750 +00:00] [ERROR] [conn.go:1031] ["connection running loop panic"] [keyspaceName=LtHBey5ok2GWLx1] [keyspaceID=5120] [conn=2004229377528693785] [lastSQL="analyze table `chsmall` . `stock`"] [err="runtime error: invalid memory address or nil pointer dereference"] [stack="github.com/pingcap/tidb/server.(*clientConn).Run.func1\n\t/go/src/github.com/pingcap/tidb/server/conn.go:1034\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:884\ngithub.com/pingcap/tidb/executor.(*ExecStmt).Exec.func1\n\t/go/src/github.com/pingcap/tidb/executor/adapter.go:475\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:884\ngithub.com/pingcap/tidb/executor.(*ExecStmt).Exec.func1\n\t/go/src/github.com/pingcap/tidb/executor/adapter.go:475\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:884\nruntime.panicmem\n\t/usr/local/go/src/runtime/panic.go:260\nruntime.sigpanic\n\t/usr/local/go/src/runtime/signal_unix.go:835\ngithub.com/tikv/pd/client/resource_group/controller.add\n\t/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20230420042919-0e0313adef1f/resource_group/controller/model.go:197\ngithub.com/tikv/pd/client/resource_group/controller.(*groupCostController).onResponse\n\t/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20230420042919-0e0313adef1f/resource_group/controller/controller.go:1106\ngithub.com/tikv/pd/client/resource_group/controller.(*ResourceGroupsController).OnResponse\n\t/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20230420042919-0e0313adef1f/resource_group/controller/controller.go:426\ngithub.com/tikv/client-go/v2/internal/client.buildResourceControlInterceptor.func1.1\n\t/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.8-0.20230426112124-fcf95952fd7a/internal/client/client_interceptor.go:116\ngithub.com/tikv/client-go/v2/internal/client.interceptedClient.SendRequest\n\t/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.8-0.20230426112124-fcf95952fd7a/internal/client/client_interceptor.go:61\ngithub.com/tikv/client-go/v2/internal/client.reqCollapse.SendRequest\n\t/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.8-0.20230426112124-fcf95952fd7a/internal/client/client_collapse.go:74\ngithub.com/tikv/client-go/v2/internal/locate.(*RegionRequestSender).sendReqToRegion\n\t/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.8-0.20230426112124-fcf95952fd7a/internal/locate/region_request.go:1391\ngithub.com/tikv/client-go/v2/internal/locate.(*RegionRequestSender).SendReqCtx\n\t/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.8-0.20230426112124-fcf95952fd7a/internal/locate/region_request.go:1209\ngithub.com/tikv/client-go/v2/internal/locate.(*RegionRequestSender).SendReq\n\t/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.8-0.20230426112124-fcf95952fd7a/internal/locate/region_request.go:238\ngithub.com/tikv/client-go/v2/txnkv/transaction.actionPessimisticLock.handleSingleBatch\n\t/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.8-0.20230426112124-fcf95952fd7a/txnkv/transaction/pessimistic.go:189\ngithub.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnBatches\n\t/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.8-0.20230426112124-fcf95952fd7a/txnkv/transaction/2pc.go:1063\ngithub.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnGroupMutations\n\t/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.8-0.20230426112124-fcf95952fd7a/txnkv/transaction/2pc.go:988\ngithub.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnMutations\n\t/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.8-0.20230426112124-fcf95952fd7a/txnkv/transaction/2pc.go:807\ngithub.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).pessimisticLockMutations\n\t/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.8-0.20230426112124-fcf95952fd7a/txnkv/transaction/pessimistic.go:613\ngithub.com/tikv/client-go/v2/txnkv/transaction.(*KVTxn).lockKeys\n\t/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.8-0.20230426112124-fcf95952fd7a/txnkv/transaction/txn.go:1090\ngithub.com/tikv/client-go/v2/txnkv/transaction.(*KVTxn).LockKeysFunc\n\t/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.8-0.20230426112124-fcf95952fd7a/txnkv/transaction/txn.go:902\ngithub.com/pingcap/tidb/store/driver/txn.(*tikvTxn).LockKeysFunc\n\t/go/src/github.com/pingcap/tidb/store/driver/txn/txn_driver.go:97\ngithub.com/pingcap/tidb/session.(*LazyTxn).LockKeysFunc\n\t/go/src/github.com/pingcap/tidb/session/txn.go:473\ngithub.com/pingcap/tidb/session.(*LazyTxn).LockKeys\n\t/go/src/github.com/pingcap/tidb/session/txn.go:448\ngithub.com/pingcap/tidb/executor.doLockKeys\n\t/go/src/github.com/pingcap/tidb/executor/executor.go:1339\ngithub.com/pingcap/tidb/executor.(*PointGetExecutor).lockKeyBase\n\t/go/src/github.com/pingcap/tidb/executor/point_get.go:421\ngithub.com/pingcap/tidb/executor.(*PointGetExecutor).lockKeyIfNeeded\n\t/go/src/github.com/pingcap/tidb/executor/point_get.go:396\ngithub.com/pingcap/tidb/executor.(*PointGetExecutor).Next\n\t/go/src/github.com/pingcap/tidb/executor/point_get.go:256\ngithub.com/pingcap/tidb/executor.Next\n\t/go/src/github.com/pingcap/tidb/executor/executor.go:324\ngithub.com/pingcap/tidb/executor.(*ExecStmt).next\n\t/go/src/github.com/pingcap/tidb/executor/adapter.go:1214\ngithub.com/pingcap/tidb/executor.(*ExecStmt).runPessimisticSelectForUpdate\n\t/go/src/github.com/pingcap/tidb/executor/adapter.go:924\ngithub.com/pingcap/tidb/executor.(*ExecStmt).handlePessimisticSelectForUpdate\n\t/go/src/github.com/pingcap/tidb/executor/adapter.go:895\ngithub.com/pingcap/tidb/executor.(*ExecStmt).Exec\n\t/go/src/github.com/pingcap/tidb/executor/adapter.go:589\ngithub.com/pingcap/tidb/session.runStmt\n\t/go/src/github.com/pingcap/tidb/session/session.go:2315\ngithub.com/pingcap/tidb/session.(*session).ExecuteStmt\n\t/go/src/github.com/pingcap/tidb/session/session.go:2169\ngithub.com/pingcap/tidb/session.(*session).ExecuteInternal\n\t/go/src/github.com/pingcap/tidb/session/session.go:1623\ngithub.com/pingcap/tidb/statistics/handle.SaveTableStatsToStorage\n\t/go/src/github.com/pingcap/tidb/statistics/handle/handle.go:1549\ngithub.com/pingcap/tidb/statistics/handle.(*Handle).SaveTableStatsToStorage\n\t/go/src/github.com/pingcap/tidb/statistics/handle/handle.go:1513\ngithub.com/pingcap/tidb/executor.(*AnalyzeExec).handleResultsError\n\t/go/src/github.com/pingcap/tidb/executor/analyze.go:318\ngithub.com/pingcap/tidb/executor.(*AnalyzeExec).Next\n\t/go/src/github.com/pingcap/tidb/executor/analyze.go:185\ngithub.com/pingcap/tidb/executor.Next\n\t/go/src/github.com/pingcap/tidb/executor/executor.go:324\ngithub.com/pingcap/tidb/executor.(*ExecStmt).next\n\t/go/src/github.com/pingcap/tidb/executor/adapter.go:1214\ngithub.com/pingcap/tidb/executor.(*ExecStmt).handleNoDelayExecutor\n\t/go/src/github.com/pingcap/tidb/executor/adapter.go:966\ngithub.com/pingcap/tidb/executor.(*ExecStmt).handleNoDelay\n\t/go/src/github.com/pingcap/tidb/executor/adapter.go:798\ngithub.com/pingcap/tidb/executor.(*ExecStmt).Exec\n\t/go/src/github.com/pingcap/tidb/executor/adapter.go:593\ngithub.com/pingcap/tidb/session.runStmt\n\t/go/src/github.com/pingcap/tidb/session/session.go:2315\ngithub.com/pingcap/tidb/session.(*session).ExecuteStmt\n\t/go/src/github.com/pingcap/tidb/session/session.go:2169\ngithub.com/pingcap/tidb/server.(*TiDBContext).ExecuteStmt\n\t/go/src/github.com/pingcap/tidb/server/driver_tidb.go:247\ngithub.com/pingcap/tidb/server.(*clientConn).handleStmt\n\t/go/src/github.com/pingcap/tidb/server/conn.go:1951\ngithub.com/pingcap/tidb/server.(*clientConn).handleQuery\n\t/go/src/github.com/pingcap/tidb/server/conn.go:1801\ngithub.com/pingcap/tidb/server.(*clientConn).dispatch\n\t/go/src/github.com/pingcap/tidb/server/conn.go:1330\ngithub.com/pingcap/tidb/server.(*clientConn).Run\n\t/go/src/github.com/pingcap/tidb/server/conn.go:1096\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/go/src/github.com/pingcap/tidb/server/server.go:631"]

What version of PD are you using (pd-server -V)?

latest pd-cse

[mayjiang0203]

/assign @nolouch