Sentry: crash_reporting.go:410: raft closed timestamp regression; replica has: ×, new batch has: ×. (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/util/log/logcr...

[cockroach-sentry]

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/6862500860/?referrer=webhooks_plugin

Panic Message:

crash_reporting.go:410: raft closed timestamp regression; replica has: ×, new batch has: ×.
(1) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/cockroach/pkg/util/log/logcrash.ReportOrPanic
  | 	github.com/cockroachdb/cockroach/pkg/util/log/logcrash/crash_reporting.go:410
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).ApplyToStateMachine
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_app_batch.go:630
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:285
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:246
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1094
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:753
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:684
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:412
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:315
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
  | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:480
  | runtime.goexit
  | 	src/runtime/asm_amd64.s:1695
Wraps: (2) secondary error attachment
  | raft closed timestamp regression; replica has: ×, new batch has: ×.
  | (1) assertion failure
  | Wraps: (2) attached stack trace
  |   -- stack trace:
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).ApplyToStateMachine
  |   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_app_batch.go:627
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch
  |   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:285
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries
  |   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:246
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
  |   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1094
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
  |   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:753
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
  |   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:684
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
  |   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:412
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
  |   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:315
  |   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
  |   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:480
  |   | runtime.goexit
  |   | 	src/runtime/asm_amd64.s:1695
  | Wraps: (3) raft closed timestamp regression; replica has: ×, new batch has: ×.
  | Error types: (1) *assert.withAssertionFailure (2) *withstack.withStack (3) *errutil.leafError
Wraps: (3) raft closed timestamp regression; replica has: ×, new batch has: ×.
Error types: (1) *withstack.withStack (2) *secondary.withSecondaryError (3) *errutil.leafError
-- report composition:
*errutil.leafError: raft closed timestamp regression; replica has: ×, new batch has: ×.
*secondary.withSecondaryError: details for github.com/cockroachdb/errors/withstack/*withstack.withStack:::
crash_reporting.go:410: *withstack.withStack (top exception)

Stacktrace (expand for inline code snippets):

src/runtime/asm_amd64.s#L1694-L1696

cockroach/pkg/util/stop/stopper.go

Lines 479 to 481 in c8459c0

	sp.UpdateGoroutineIDToCurrent()
	f(ctx)
	}()

https://github.com/cockroachdb/cockroach/blob/c8459c085e2258f589949e970a527a0342871276/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L314-L316
https://github.com/cockroachdb/cockroach/blob/c8459c085e2258f589949e970a527a0342871276/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L411-L413
https://github.com/cockroachdb/cockroach/blob/c8459c085e2258f589949e970a527a0342871276/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L683-L685
https://github.com/cockroachdb/cockroach/blob/c8459c085e2258f589949e970a527a0342871276/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L752-L754
https://github.com/cockroachdb/cockroach/blob/c8459c085e2258f589949e970a527a0342871276/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L1093-L1095

cockroach/pkg/kv/kvserver/apply/task.go

Lines 245 to 247 in c8459c0

	for iter.Valid() {
	if err := t.applyOneBatch(ctx, iter); err != nil {
	// If the batch threw an error, reject all remaining commands in the

cockroach/pkg/kv/kvserver/apply/task.go

Lines 284 to 286 in c8459c0

	// Apply the persistent state transitions to the state machine.
	if err := batch.ApplyToStateMachine(ctx); err != nil {
	return err

https://github.com/cockroachdb/cockroach/blob/c8459c085e2258f589949e970a527a0342871276/pkg/kv/kvserver/pkg/kv/kvserver/replica_app_batch.go#L629-L631

cockroach/pkg/util/log/logcrash/crash_reporting.go

Lines 409 to 411 in c8459c0

	) {
	err := errors.Newf(format, reportables...)
	if !build.IsRelease() || (sv != nil && PanicOnAssertions.Get(sv)) {

src/runtime/asm_amd64.s in runtime.goexit at line 1695
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 480
pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftScheduler).Start.func2 at line 315
pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftSchedulerShard).worker at line 412
pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processReady at line 684
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReady at line 753
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked at line 1094
pkg/kv/kvserver/apply/task.go in pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries at line 246
pkg/kv/kvserver/apply/task.go in pkg/kv/kvserver/apply.(*Task).applyOneBatch at line 285
pkg/kv/kvserver/pkg/kv/kvserver/replica_app_batch.go in pkg/kv/kvserver.(*replicaAppBatch).ApplyToStateMachine at line 630
pkg/util/log/logcrash/crash_reporting.go in pkg/util/log/logcrash.ReportOrPanic at line 410

Tags

Tag	Value
Command	server
Environment	v24.1.14
Go Version	go1.22.5 X:nocoverageredesign
Platform	linux amd64
Distribution	CCL
Cockroach Release	v24.1.14
Cockroach SHA	c8459c0
# of CPUs	64
# of Goroutines	792

Jira issue: CRDB-54215

[blathers-crl]

CC'ing via the CODEOWNERS-based sentry heuristic:

• @cockroachdb/obs-prs

Sentry issue cause: pkg/util/log/logcrash/crash_reporting.go

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}