kvserver: sep-raft-log: make log application stand-alone

[tbg]

Is your feature request related to a problem? Please describe.

Today, there is no "library-style" way of crafting a CockroachDB log of commands.

We can in principle craft individual commands by evaluating requests (the
batcheval package), but below that you need a *Replica which strings
together the raft log, and the apply loop.

We have felt for a long time that abstractly we could benefit from better
modularization here, but now we are starting to see this come up in projects we
are working on, for example

cockroach/pkg/storage/replicas_storage.go

Lines 133 to 142 in 4a73be0

	// ReplicasStorage does interpret the raft state (all the unreplicated
	// range-ID local key names prefixed by Raft), and the RangeTombstoneKey. This
	// is necessary for it to be able to maintain invariants spanning the raft log
	// and the state machine (related to raft log truncation, replica lifetime
	// etc.), including reapplying raft log entries on restart, to the state
	// machine. All accesses (read or write) to the raft log and RangeTombstoneKey
	// must happen via ReplicasStorage. ReplicasStorage does not by itself apply
	// committed raft log entries to the state machine in a running system -- this
	// is because state machine application in a running system has complex
	// data-structure side-effects that are outside the scope of ReplicasStorage.

and similarly Loss Of Quorum Recovery tools could benefit from tools to interpret the raft log at the right level of abstraction.

Describe the solution you'd like

Continue the work begun by #39254
and further extract the logic pertaining to entry application so that it can be
used to apply log entries outside of the context of a fully instantiated
Replica. I think most of the work will be to wean the apply code off the
many direct accesses to replica.mu and the associated fields.

As an acceptance criterion, we should be able to write a unit test that sets up
a test cluster, appends some commands to the log, stops the cluster and
subsequently re-applies the log entries to an empty state, arriving at an
identical state machine.

Ideally, we also get a testing win by allowing tests to be set up with bespoke
raft logs that exercise hand-crafted sequences of commands; this is very difficult
today and essentially doesn't happen for that very reason.

Note: inclusion of splits/merges will add complexity. We may not need this in
the first pass but should think about how this could work in the future.

Epic: CRDB-220
Jira issue: CRDB-12807

[tbg]

[tbg]

I'm not sure this photo of my whiteboard is helpful for anyone but me, but I spent a little bit of time looking into what's needed here. I think there are a few "themes":

• the "log encoding" portion on the right-hand side is done manually in multiple places today. A little abstraction over this would be useful and this should be straightforward to write as well.
• the apply-side (left hand side) is already nicely modular thanks to prior work by @nvanbenschoten (all the way back in '19!), but lots of the pieces of code that would need to be reusable aren't quite yet, so we need to pull out things like checkForcedError to make them re-usable. I expect most of these to be surgical.
• I don't think we're too far off from the evaluate-side either. Assuming everything the commands we want to manufacture programmatically is populated in the batcheval.EvalContext, we will get a RaftCommand back and making the lease/closed timestamp code reusable shouldn't be hard either, especially since this is all much easier when operating offline without concurrency or in-memory state. We don't care much about the eval side outside of testing, so supporting just a subset of commands when we need it is good enough.
• to support splits and merges, we would similarly need to make all the disk-touching components of these operations' side effects reusable, and probably restructure the code a bit to make sure there aren't two parallel control flows that can diverge easily. It's unclear if we need to support those anytime soon, just putting it out there.

[tbg]

The right side of the picture ("Log encoding") is librarified, at least on the read path, in #76126. We still "manually" create these proposals when we run config changes:

cockroach/pkg/kv/kvserver/replica_proposal_buf.go

Lines 529 to 559 in 7051448

	// Coordinate proposing the command to etcd/raft.
	if crt := p.command.ReplicatedEvalResult.ChangeReplicas; crt != nil {
	// Flush any previously batched (non-conf change) proposals to
	// preserve the correct ordering or proposals. Later proposals
	// will start a new batch.
	if err := proposeBatch(raftGroup, b.p.replicaID(), ents); err != nil {
	firstErr = err
	continue
	}
	ents = ents[len(ents):]
	confChangeCtx := kvserverpb.ConfChangeContext{
	CommandID: string(p.idKey),
	Payload: p.encodedCommand,
	}
	encodedCtx, err := protoutil.Marshal(&confChangeCtx)
	if err != nil {
	firstErr = err
	continue
	}
	cc, err := crt.ConfChange(encodedCtx)
	if err != nil {
	firstErr = err
	continue
	}
	if err := raftGroup.ProposeConfChange(
	cc,
	); err != nil && !errors.Is(err, raft.ErrProposalDropped) {
	// Silently ignore dropped proposals (they were always silently

[sumeerbhola]

I don't think we're too far off from the evaluate-side either. Assuming everything the commands we want to manufacture programmatically is populated in the batcheval.EvalContext, we will get a RaftCommand back and making the lease/closed timestamp code reusable shouldn't be hard either, especially since this is all much easier when operating offline without concurrency or in-memory state. We don't care much about the eval side outside of testing, so supporting just a subset of commands when we need it is good enough.

We don't need this for ReplicasStorage, and I'm guessing we don't need it for loss of quorum either. Is that correct? Why do we need to write tests that need to start with a BatchRequest and don't have a Replica (just trying to understand)? Isn't populating batcheval.EvalContext a tall order given the very broad interface?

[tbg]

This isn't tracking exactly the work needed for ReplicasStorage (which will only need to apply some kinds of entries excluding splits and merges, iirc). It's generally what we want to exist for replication in a few places, and while there isn't a super pressing need, being able to do this shows us that we're doing something right.

[tbg]

The concrete example where you might want it both for LOQ and for ReplicasStorage is that for testing you'll want to populate custom raft logs. Having to spin up servers and manufacture them in a live system is now how this should work. Yet, that's what we do today:

cockroach/pkg/kv/kvserver/loqrecovery/collect_raft_log_test.go

Lines 34 to 90 in c4d4df9

	// TestFindUpdateDescriptor verifies that we can detect changes to range
	// descriptor in the raft log.
	// To do this we split and merge the range which updates descriptor prior to
	// spawning or subsuming RHS.
	func TestFindUpdateDescriptor(t *testing.T) {
	defer leaktest.AfterTest(t)()
	defer log.Scope(t).Close(t)
	ctx := context.Background()
	const testNode = 0
	var testRangeID roachpb.RangeID
	var rHS roachpb.RangeDescriptor
	var lHSBefore roachpb.RangeDescriptor
	var lHSAfter roachpb.RangeDescriptor
	checkRaftLog(t, ctx, testNode,
	func(ctx context.Context, tc *testcluster.TestCluster) roachpb.RKey {
	scratchKey, err := tc.Server(0).ScratchRange()
	require.NoError(t, err, "failed to get scratch range")
	srk, err := keys.Addr(scratchKey)
	require.NoError(t, err, "failed to resolve scratch key")
	rd, err := tc.LookupRange(scratchKey)
	testRangeID = rd.RangeID
	require.NoError(t, err, "failed to get descriptor for scratch range")
	splitKey := testutils.MakeKey(scratchKey, []byte("z"))
	lHSBefore, rHS, err = tc.SplitRange(splitKey)
	require.NoError(t, err, "failed to split scratch range")
	lHSAfter, err = tc.Servers[0].MergeRanges(scratchKey)
	require.NoError(t, err, "failed to merge scratch range")
	require.NoError(t,
	tc.Server(testNode).DB().Put(ctx, testutils.MakeKey(scratchKey, []byte("|first")),
	"some data"),
	"failed to put test value in LHS")
	return srk
	},
	func(t *testing.T, ctx context.Context, reader storage.Reader) {
	seq, err := loqrecovery.GetDescriptorChangesFromRaftLog(testRangeID, 0, math.MaxInt64, reader)
	require.NoError(t, err, "failed to read raft log data")
	requireContainsDescriptor(t, loqrecoverypb.DescriptorChangeInfo{
	ChangeType: loqrecoverypb.DescriptorChangeType_Split,
	Desc: &lHSBefore,
	OtherDesc: &rHS,
	}, seq)
	requireContainsDescriptor(t, loqrecoverypb.DescriptorChangeInfo{
	ChangeType: loqrecoverypb.DescriptorChangeType_Merge,
	Desc: &lHSAfter,
	OtherDesc: &rHS,
	}, seq)
	})
	}

[tbg]

In #97347 we're seeing a tricky interaction between reproposals that we can't unit test without a significant rework. Having more stand-alone components for log application would help here.