storage: re-enqueue Raft groups on paginated application

[nvanbenschoten]

Fixes #31330.

This change re-enqueues Raft groups for processing immediately if they
still have more to do after a Raft ready iteration. This comes up in
practice when a Range has sufficient load to force Raft application
pagination. See #31330 for a discussion on the symptoms this can
cause.

Release note (bug fix): Fix bug where Raft followers could fall behind
leaders will entry application, causing stalls during splits.

[cockroach-teamcity]

This change is 

[cockroach-teamcity]

This change is 

[petermattis]

Have you stressed this new test? Mucking with proposal application like that seems fragile. I don't have a better suggestion, though.

Reviewable status: complete! 0 of 0 LGTMs obtained (and 1 stale)

---

pkg/storage/replica_test.go, line 9622 at r1 (raw file):

	repl := tc.repl

	// Propose a command to Raft and block its application.

Technically, you block application and then propose the command.

[nvanbenschoten]

TFTR.

Have you stressed this new test?

I stressed it for 10 minutes without issue.

Reviewable status: complete! 0 of 0 LGTMs obtained (and 1 stale)

---

pkg/storage/replica_test.go, line 9622 at r1 (raw file):

Previously, petermattis (Peter Mattis) wrote…

Technically, you block application and then propose the command.

Done.

[tschottdorf]

Reviewed 1 of 2 files at r1, 1 of 1 files at r2.
Reviewable status: complete! 0 of 0 LGTMs obtained (and 2 stale)

---

pkg/storage/replica_test.go, line 9586 at r2 (raw file):

// TestApplyPaginatedCommittedEntries tests that a Raft group's committed
// entries are quickly applied, even if their application is paginated due to
// the RaftMaxSizePerMsg configuration. This is a regession test for #31330.

nit: regession

---

pkg/storage/replica_test.go, line 9659 at r2 (raw file):

	// small RaftMaxSizePerMsg.
	close(blockRaftApplication)
	const maxWait = 5 * time.Second

5s might be too aggressive to apply 50 commands when stressed sufficiently. Perhaps bump this a bit. Or we'll just leave it and find out.