kvserver: recompute stats after mvcc gc

[lunevalex]

Touched #82920

There is at least one known issue in MVCC stats calculation and
there maybe more. This could lead to the MVCC GC Queue spinning on
ranges with bad stats. To prevent the queue from spinning it should
recompute the stats if it detects that they are wrong. The easiest
mechanism to do that is to check if the GC score wants to queue this
range again after finishing GC, if it does it likely indicates something
fishy with the stats.

Release note: Change the MVCC GC queue to recompute MVCC stats on a
range, if after doing a GC run it still thinks there is garbage in
the range.

[cockroach-teamcity]

This change is 

[lunevalex]

@erikgrinaker I cleaned this up and added a test, do you want to take a look?

[erikgrinaker]

Thanks for handling this! Might deserve a brief release note?

Reviewed 3 of 3 files at r1, all commit messages.
Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @lunevalex)

---

pkg/kv/kvserver/mvcc_gc_queue.go line 616 at r1 (raw file):

	scoreAfter := makeMVCCGCQueueScore(
		ctx, repl, repl.store.Clock().Now(), lastGC, conf.TTL(), canAdvanceGCThreshold)

We're reusing the old lastGC and canAdvanceGCThreshold here, so the score/decision isn't what it would be if the GC queue reprocessed it. That seems mostly beneficial, since it avoids issues with e.g. cooldown timers, but it might be worth calling out.

---

pkg/kv/kvserver/mvcc_gc_queue.go line 627 at r1 (raw file):

	// them then to spin the GC queue.
	if scoreAfter.ShouldQueue {
		log.Infof(ctx, "triggering stats re-computation")

Would be good to elaborate slightly here, and include the previous and current GC scores for debugging. Something like "GC still needed following GC, recomputing MVCC stats (old score %s; new score %s)". The score formatting is pretty verbose though, so see how it looks and consider dumping it on multiple lines or sth.

---

pkg/kv/kvserver/mvcc_gc_queue.go line 635 at r1 (raw file):

		err := repl.store.db.Run(ctx, &b)
		if err != nil {
			log.Errorf(ctx, "Failed to recompute stats with error=%s", err)

nit: start with lowercase.

[lunevalex]

TFTR!

bors r+

[craig]

Build succeeded:

• GitHub CI (Cockroach)