pd panic after fault recover from time offset 5mins

[Lily2025]

Bug Report

What did you do?

1、run tpcc （warehouse 2w）with 32 threads
2、inject pd leader time offset 5mins
3、after 10mins，recover fault

What did you expect to see?

no panic

What did you see instead?

after 5mins when fault recover，pd panic
2024-04-09 11:30:20 log="/pd-server --data-dir=/var/lib/pd/data --name=tc-pd-1 --peer-urls=http://0.0.0.0:2380 --advertise-peer-urls=http://tc-pd-1.tc-pd-peer.ha-test-lightning-tps-7503715-1-618.svc:2380 --client-urls=http://0.0.0.0:2379 --advertise-client-urls=http://tc-pd-1.tc-pd-peer.ha-test-lightning-tps-7503715-1-618.svc:2379 --config=/etc/pd/pd.toml \n" 2024-04-09 11:30:16 log="starting pd-server ...\n" 2024-04-09 11:30:16 log="nslookup domain tc-pd-1.tc-pd-peer.ha-test-lightning-tps-7503715-1-618.svc.svc success\n" 2024-04-09 11:30:16 log="\n" 2024-04-09 11:30:16 log="Address: 10.233.88.210\n" 2024-04-09 11:30:16 log="Name:\ttc-pd-1.tc-pd-peer.ha-test-lightning-tps-7503715-1-618.svc.cluster.local\n" 2024-04-09 11:30:16 log="\n" 2024-04-09 11:30:16 log="Address:\t10.96.0.10#53\n" 2024-04-09 11:30:16 log="Server:\t\t10.96.0.10\n" 2024-04-09 11:30:14 log="\t/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1038 +0x135\n" 2024-04-09 11:30:14 log="created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 1888642\n" 2024-04-09 11:30:14 log="\t/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1027 +0x8b\n" 2024-04-09 11:30:14 log="google.golang.org/grpc.(*Server).serveStreams.func2.1()\n" 2024-04-09 11:30:14 log="\t/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1801 +0xfbb\n" 2024-04-09 11:30:14 log="google.golang.org/grpc.(*Server).handleStream(0xc0015f0c00, {0x3c56cc0, 0xc005c5fa00}, 0xc0070ca5a0)\n" 2024-04-09 11:30:14 log="\t/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1687 +0x1267\n" 2024-04-09 11:30:14 log="google.golang.org/grpc.(*Server).processStreamingRPC(0xc0015f0c00, {0x3c49788, 0xc00667b770}, {0x3c56cc0, 0xc005c5fa00}, 0xc0070ca5a0, 0xc00124fe90, 0x5086f20, 0x0)\n" 2024-04-09 11:30:14 log="\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/chain.go:86 +0x135\n" 2024-04-09 11:30:14 log="go.etcd.io/etcd/etcdserver/api/v3rpc.Server.ChainStreamServer.func9({0x2ff7a80, 0xc001786ae0}, {0x3c4f060, 0xc004c51860}, 0xc00d202d98, 0xc00f561c40?)\n" 2024-04-09 11:30:14 log="\t/go/pkg/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20240320135013-950cd5fbe6ca/etcdserver/api/v3rpc/interceptor.go:238 +0x479\n" 2024-04-09 11:30:14 log="go.etcd.io/etcd/etcdserver/api/v3rpc.newStreamInterceptor.func1({0x2ff7a80, 0xc001786ae0}, {0x3c4f060, 0xc004c51860}, 0xc00d202d98, 0xc0070a9a00)\n" 2024-04-09 11:30:14 log="\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/chain.go:83 +0x45\n" 2024-04-09 11:30:14 log="go.etcd.io/etcd/etcdserver/api/v3rpc.Server.ChainStreamServer.func9.1({0x2ff7a80?, 0xc001786ae0?}, {0x3c4f060?, 0xc004c51860?})\n" 2024-04-09 11:30:14 log="\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-prometheus@v1.2.0/server_metrics.go:121 +0xd2\n" 2024-04-09 11:30:14 log="github.com/grpc-ecosystem/go-grpc-prometheus.init.(*ServerMetrics).StreamServerInterceptor.func4({0x2ff7a80, 0xc001786ae0}, {0x3c4f060?, 0xc004c51860}, 0xc00d202d98, 0x39716b8)\n" 2024-04-09 11:30:14 log="\t/go/pkg/mod/github.com/pingcap/kvproto@v0.0.0-20240403065636-c699538f7aa1/pkg/pdpb/pdpb.pb.go:9777 +0x94\n" 2024-04-09 11:30:14 log="github.com/pingcap/kvproto/pkg/pdpb._PD_RegionHeartbeat_Handler({0x2ff7a80?, 0xc001786ae0}, {0x3c4e7f0?, 0xc00d202db0})\n" 2024-04-09 11:30:14 log="\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/server/grpc_service.go:1296 +0xf72\n" 2024-04-09 11:30:14 log="github.com/tikv/pd/server.(*GrpcServer).RegionHeartbeat(0xc001786ae0, {0x3c556d0?, 0xc00f561c90})\n" 2024-04-09 11:30:14 log="\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/server/cluster/cluster_worker.go:51 +0x1af\n" 2024-04-09 11:30:14 log="github.com/tikv/pd/server/cluster.(*RaftCluster).HandleRegionHeartbeat(0xc00056b680, 0xc00388cdd0?)\n" 2024-04-09 11:30:14 log="\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/pkg/schedule/operator/operator_controller.go:159 +0x875\n" 2024-04-09 11:30:14 log="github.com/tikv/pd/pkg/schedule/operator.(*Controller).Dispatch(0xc00fa80000, 0xc00eb05300, {0x300c8c2, 0x9}, 0xc00c38ad70)\n" 2024-04-09 11:30:14 log="\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/pkg/schedule/operator/operator_controller.go:382 +0x3d\n" 2024-04-09 11:30:14 log="github.com/tikv/pd/pkg/schedule/operator.(*Controller).PromoteWaitingOperator(0xc00fa80000)\n" 2024-04-09 11:30:14 log="\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/pkg/schedule/operator/waiting_operator.go:101 +0x365\n" 2024-04-09 11:30:14 log="github.com/tikv/pd/pkg/schedule/operator.(*randBuckets).GetOperator(0xc00c7df8c0)\n" 2024-04-09 11:30:14 log="goroutine 1888645 [running]:\n" 2024-04-09 11:30:14 log="\n" 2024-04-09 11:30:14 log="panic: runtime error: index out of range [1] with length 1\n"

What version of PD are you using (pd-server -V)?

./pd-server -V
Release Version: v8.1.0-alpha
Edition: Community
Git Commit Hash: 726b81f
Git Branch: heads/refs/tags/v8.1.0-alpha
UTC Build Time: 2024-04-08 11:37:36
2024-04-09T01:44:40.309+0800
./tidb-server -V
Release Version: v8.1.0-alpha

[Lily2025]

/type bug
/severity critical

[rleungx]

Previously, we added two merge operators at the same time. After #8032, we might add two merge operators one by one. If we promote the merge operator before adding the second one. It might panic.