storage: smarten GC of orphaned replicas of subsumed ranges

[benesch]

When a range is subsumed, there are two paths by which its replicas can
be cleaned up. The first path is that the subsuming replica, when it
applies the merge trigger, removes the subsumed replica. This is the
common case, as all replicas are collocated when the merge transaction
starts.

The second path is that the subumed replica is later cleaned up by the
replica GC queue. This occurs when the subsuming range is rebalanced
away shortly after the merge and so never applies the merge trigger,
"orphaning" the subsuming replica.

The replica GC queue must be careful to never to GC a replica that could
be subsumed. If it discovers that a merge occurred, it needs to "prove"
that the replica is actually orphaned. It does so by checking whether
the left neighbor's local descriptor matches the meta2 descriptor; if it
does not, the left neighbor is out of date and could possibly still
apply a merge trigger, so the replica cannot be GC'd.

Unfortunately, the replica GC queue tried to be too clever: it assumed
such a proof was not necessary if the store was still a member of the
subsuming range. Concretely, suppose adjacent ranges A and B merge, and
store 2's replica of B is orphaned. When the replica GC queue looks
up B's descriptor in meta2, it will get the descriptor for the combined
range AB instead and correctly infer that a merge occurred. It also
assumed that, because AB is listed as having a replica on store2, that
the merge must be applying soon.

This assumption was wrong. Suppose the merged range AB immediately
splits back into A and B. The replica GC queue, considering store 2's
replica of the new B, will, again, correctly infer that a merge took
place (even though the descriptor it fetches from meta2 will have the
same start and end key as its local descriptor, it will have a new range
ID), but now its assumption that a replica of A must exist on the store
is incorrect! A may have been rebalanced away, in which case we must
GC the old copy of B, or the store will never be able to accept a
snapshot for the new copy of B.

This scenario was observed in several real clusters, and easily
reproduces when restoring TPC-C.

The fix is simple: teach the replica GC queue to always perform the
proof when a range has been merged away. Attempting to be clever just to
save one meta2 lookup was a bad idea.

Touches #31409.

Release note: None

/cc @andreimatei

[cockroach-teamcity]

This change is 

[cockroach-teamcity]

This change is 

[tschottdorf]

:

Reviewed 2 of 2 files at r1.
Reviewable status: complete! 1 of 0 LGTMs obtained

---

pkg/storage/replica_gc_queue.go, line 264 at r1 (raw file):

			}
			if leftReplyDesc := rs[0]; !leftDesc.Equal(leftReplyDesc) {
				if log.V(1) {

While you're here, can you make all the V(x)s here into VEventfs so that Alex' queue endpoint gets better output?