kvserver: lease transferred to follower waiting for split

[tbg]

Describe the problem

In an experiment, a ten node cluster was run with IO overload on n3, and a 2TB bank import. This caused many of the splits involving a follower on n3 to get into a state where the n3 replica was "uninitialized" (because the split trigger would be wildly delayed.

We were seeing evidence that sometimes, the lease would get transferred to the n3 replica, creating an outage. SSTs would then get stuck in NotleaseholderError loops and bounce around for hours.

The full internal thread is here

To Reproduce

Set up 10 node AWS cluster via roachprod according to steps in https://cockroachlabs.slack.com/archives/C0KB9Q03D/p1649015732041819 and deploy the following unit to n3:

# Throughput killer
roachprod ssh tobias-import:3 -- sudo systemd-run --unit fiotp --working-directory=/mnt/data1/ -- fio --rw=readwrite --name=test --size=50M --direct=1 --bs=1024k --ioengine=libaio --iodepth=16 --directory=/mnt/data1/ --time_based --timeout 2400h

Ran this on
71e32a6

Expected behavior

We don't transfer leases to replicas that have never been initialized.

Environment:
master on roachprod AWS

Jira issue: CRDB-14780

Epic CRDB-16160

[tbg]

cc @cockroachdb/kv

[tbg]

Probably the store rebalancer; I don't think it logs individual transfers

$ ./cockroach debug merge-logs logs/cockroach.ip* | grep store_rebalancer | grep '220404 13:5[5678]'
ip-10-12-25-66> I220404 13:55:39.731276 238 kv/kvserver/pkg/kv/kvserver/store_rebalancer.go:290 ⋮ [n8,s8,store-rebalancer] 30787 local QPS 2.06 is below max threshold 226.57 (mean=126.57); no rebalancing needed
ip-10-12-25-66> I220404 13:56:30.878693 238 kv/kvserver/pkg/kv/kvserver/store_rebalancer.go:299 ⋮ [n8,s8,store-rebalancer] 30817 considering load-based lease transfers for s8 with 390.22 qps (mean=84.51, upperThreshold=184.51)
ip-10-12-25-66> I220404 13:56:30.922601 238 kv/kvserver/pkg/kv/kvserver/store_rebalancer.go:349 ⋮ [n8,s8,store-rebalancer] 30818 ran out of leases worth transferring and qps (387.84) is still above desired threshold (184.51); considering load-based replica rebalances
ip-10-12-25-66> I220404 13:56:30.922827 238 kv/kvserver/pkg/kv/kvserver/store_rebalancer.go:366 ⋮ [n8,s8,store-rebalancer] 30819 ran out of replicas worth transferring and qps (387.84) is still above desired threshold (184.51); will check again soon
ip-10-12-25-66> I220404 13:57:24.490635 238 kv/kvserver/pkg/kv/kvserver/store_rebalancer.go:299 ⋮ [n8,s8,store-rebalancer] 30833 considering load-based lease transfers for s8 with 221.99 qps (mean=29.11, upperThreshold=129.11)
ip-10-12-25-66> I220404 13:57:24.493504 238 kv/kvserver/pkg/kv/kvserver/store_rebalancer.go:349 ⋮ [n8,s8,store-rebalancer] 30834 ran out of leases worth transferring and qps (221.74) is still above desired threshold (129.11); considering load-based replica rebalances
ip-10-12-25-66> I220404 13:57:24.493701 238 kv/kvserver/pkg/kv/kvserver/store_rebalancer.go:366 ⋮ [n8,s8,store-rebalancer] 30835 ran out of replicas worth transferring and qps (221.74) is still above desired threshold (129.11); will check again soon
ip-10-12-25-66> I220404 13:58:13.499267 238 kv/kvserver/pkg/kv/kvserver/store_rebalancer.go:299 ⋮ [n8,s8,store-rebalancer] 30850 considering load-based lease transfers for s8 with 1751.70 qps (mean=214.14, upperThreshold=314.14)
ip-10-12-25-66> I220404 13:58:13.499421 238 kv/kvserver/pkg/kv/kvserver/store_rebalancer.go:349 ⋮ [n8,s8,store-rebalancer] 30851 ran out of leases worth transferring and qps (1751.70) is still above desired threshold (314.14); considering load-based replica rebalances
ip-10-12-25-66> I220404 13:58:13.499434 238 kv/kvserver/pkg/kv/kvserver/store_rebalancer.go:366 ⋮ [n8,s8,store-rebalancer] 30852 ran out of replicas worth transferring and qps (1751.70) is still above desired threshold (314.14); will check again soon

[nvanbenschoten]

This is related to #81561. I don't think such a lease transfer would be allowed if our protection against lease transfers to replicas that may need a snapshot was airtight. The replica here would be classified as potentially needing a snapshot because it has not yet performed its split and so it could not be in StateReplicate. #81764 would also allow us to recover from the faulty lease transfer in 9s even if it was allowed, instead of blocking indefinitely until the split is applied.