storage: pull Store-level concurrency retry loop under Replica, clean up req validation #43138

nvanbenschoten · 2019-12-12T23:31:29Z

This PR contains a number of cleanup steps necessary to pave the way for the unified pkg/storage/concurrency package that we plan to introduce for #41720 (see prototype). Primarily, the PR cleans up Replica-level request validation, unifies some of the request synchronization logic between the read-only and read-write execution paths, and pulls the Store-level concurrency retry loop under the Replica. Future commits will pull parts of this retry loop down into the concurrency package's "concurrency manager".

cockroach-teamcity · 2019-12-12T23:31:41Z

This change is

tbg

Reviewed 3 of 3 files at r1, 10 of 10 files at r2, 2 of 2 files at r3, 3 of 3 files at r4, 1 of 1 files at r5, 1 of 1 files at r6.
Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @ajwerner, @nvanbenschoten, @sumeerbhola, and @tbg)

pkg/storage/replica.go, line 1022 at r2 (raw file):

// checkExecutionCanProceed returns an error if a batch request cannot be
// executed by the Replica. An error indicates that the Replica is not live and
// able to serve traffic or that the request is not compatable with the state of

compatible

pkg/storage/replica.go, line 1028 at r2 (raw file):

// used to indicate whether the caller has acquired latches and checked the
// Range lease. The method will only check for a pending merge if both of these
// conditions are true.

Add that lg and st can be nil and under which conditions the caller would do that.

pkg/storage/replica.go, line 1054 at r2 (raw file):

}

// checkExecutionCanProceedForRangeFeed .returns an error if a rangefeed request

extra dot

pkg/storage/replica.go, line 1074 at r2 (raw file):

}

// checkSpanInRange returns an error if a request (identified by its

nit: name is off

pkg/storage/replica_raft.go, line 68 at r4 (raw file):

func (r *Replica) evalAndPropose(
	ctx context.Context,
	lease *roachpb.Lease,

It's fine to leak the lease pointer out of r.mu.state out from under the lock because we only ever replace that pointer, but never mutate into it, right? Might be worth a comment on ReplicaState.

pkg/storage/replica_write.go, line 122 at r1 (raw file):

	// Verify that the batch can be executed.
	if ec.lg != nil {
		if err := r.checkForPendingMerge(ctx, ba, ec.lg, &status); err != nil {

Isn't is potentially problematic to order this after redirectOnOrAcquireLease? That method may try to get the lease and in doing so, may stall (waiting for the intent on the range desc to go away). I don't know if there's anything wrong with the code per se, but it does seem that there's some thinking to be had about deadlocks. It might be safer to put this ahead of the lease check.

pkg/storage/replica_write.go, line 158 at r2 (raw file):

	// Checking the context just before proposing can help avoid ambiguous errors.
	if err := ctx.Err(); err != nil {
		log.Warningf(ctx, "%s before proposing: %s", err, ba.Summary())

not new code, but this shouldn't really be a warning, should it?

This commit shuffles around the read-only Replica execution path to acquire latches before checking the Replica's lease. This makes the read-only path more like the write path and will allow for future simplification. In order to permit this, the commit needed to pull out the logic to check for a pending merge from beginCmds. This allowed us to clean it up a bit and make its interaction with other checks more clear. Release note: None

nvanbenschoten

Thanks for giving this a pass!

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @ajwerner, @sumeerbhola, and @tbg)

pkg/storage/replica.go, line 1022 at r2 (raw file):