Sentry: panic.go:770: panic: must not be holding latches (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/panic.go:770 | github.com/cockroachdb/cockroach/pkg/ut...

[cockroach-sentry]

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/7587502303/?referrer=webhooks_plugin

Panic Message:

panic.go:770: panic: must not be holding latches
(1) attached stack trace
  -- stack trace:
  | runtime.gopanic
  | 	GOROOT/src/runtime/panic.go:770
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover
  | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:226
  | runtime.gopanic
  | 	GOROOT/src/runtime/panic.go:770
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes.func1
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:104
  | runtime.gopanic
  | 	GOROOT/src/runtime/panic.go:770
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:302
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:239
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:453
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:177
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:188
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:197
  | github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal
  | 	github.com/cockroachdb/cockroach/pkg/server/node.go:1518
  | github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch
  | 	github.com/cockroachdb/cockroach/pkg/server/node.go:1719
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:703
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12
  | 	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:92
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:814
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:165
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:814
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:132
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
  | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:332
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:130
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:814
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:713
  | github.com/cockroachdb/cockroach/pkg/rpc.NewContext.ClientInterceptor.func8
  | 	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:223
  | github.com/cockroachdb/cockroach/pkg/rpc.getChainUnaryInvoker.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:898
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func3
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:783
  | github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:906
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:209
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:187
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2752
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2200
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1749
Wraps: (2) panic: must not be holding latches
  | this is tracked in github.com/cockroachdb/cockroach/issues/77663; please comment if seen
  | eval_kind=0, holding_latches=true, branch=1, first_iteration=true, stack=
  | goroutine 3675391 [running]:
  | runtime/debug.Stack()
  | 	GOROOT/src/runtime/debug/stack.go:24 +0x5e
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard(0xc027fd7d60, {0x8370af0, 0xc043912150}, 0xc0a00d4750, 0x1)
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:305 +0xae8
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq(0xc027fd7d60, {0x8370af0, 0xc043912150}, 0x5f835c0?, {0xc02f9bc120, {0x18be3779b2918da4, 0x0}, 0x0, 0x0, 0x0, ...}, ...)
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:239 +0x2f2
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc0401b1108, {0x8370af0, 0xc043912150}, 0xc02f9aa8c0, 0x6e79c88)
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:453 +0x34c
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes(0xc0401b1108, {0x8370af0?, 0xc043912120?}, 0xc02f9aa8c0)
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:177 +0x6d0
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes(0xc002bd8008, {0x8370af0?, 0xc0439120f0?}, 0xc02f9aa8c0)
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:188 +0x83c
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes(0xc0018f5180, {0x8370af0, 0xc0439120f0}, 0xc02f9aa8c0)
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:197 +0x7e
  | github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc007142c08, {0x8370af0?, 0xc0439120c0?}, {0xc035daff10?}, 0xc02f9aa8c0)
  | 	github.com/cockroachdb/cockroach/pkg/server/node.go:1518 +0x5b4
  | github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc007142c08, {0x8370af0, 0xc043912060}, 0xc02f9aa8c0)
  | 	github.com/cockroachdb/cockroach/pkg/server/node.go:1719 +0x2a6
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1({0x8370af0?, 0xc043912060?}, {0x695a960?, 0xc02f9aa8c0?})
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:703 +0x45
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12({0x8370af0, 0xc043912060}, {0x695a960, 0xc02f9aa8c0}, 0xc00eeb0480, 0xc00edff620)
  | 	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:92 +0x4b2
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0x8370af0?, 0xc043912060?}, {0x695a960?, 0xc02f9aa8c0?})
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:814 +0x34
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3({0x8370af0, 0xc043912060}, {0x695a960, 0xc02f9aa8c0}, 0xc035db0648?, 0xc00eeb04a0)
  | 	github.com/cock...

Stacktrace (expand for inline code snippets):

cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go

Lines 1748 to 1750 in 046116a

	if !ri.NeedAnother(rs) {
	resp := ds.sendPartialBatch(
	ctx, ba, rs, isReverse, withCommit, batchIdx, ri.Token(),

cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go

Lines 2199 to 2201 in 046116a

	prevTok = routingTok
	reply, err = ds.sendToReplicas(ctx, ba, routingTok, withCommit)

cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go

Lines 2751 to 2753 in 046116a

	} else {
	br, err = transport.SendNext(sendCtx, requestToSend)
	tEnd := timeutil.Now()

cockroach/pkg/kv/kvclient/kvcoord/transport.go

Lines 186 to 188 in 046116a

	}
	return gt.sendBatch(ctx, r.NodeID, iface, ba)
	}

cockroach/pkg/kv/kvclient/kvcoord/transport.go

Lines 208 to 210 in 046116a

	log.VEvent(ctx, 2, "sending batch request")
	reply, err := iface.Batch(ctx, ba)
	log.VEvent(ctx, 2, "received batch response")

https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/rpc/pkg/rpc/context.go#L905-L907
https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/rpc/pkg/rpc/context.go#L782-L784
https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/rpc/pkg/rpc/context.go#L897-L899

cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go

Lines 222 to 224 in 046116a

	if localRequest {
	return invoker(ctx, method, req, resp, cc, opts...)
	}

https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/rpc/pkg/rpc/context.go#L712-L714
https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/rpc/pkg/rpc/context.go#L813-L815
https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/rpc/pkg/rpc/context.go#L129-L131

cockroach/pkg/util/stop/stopper.go

Lines 331 to 333 in 046116a

	return f(ctx)
	}

https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/rpc/pkg/rpc/context.go#L131-L133
https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/rpc/pkg/rpc/context.go#L813-L815
https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/rpc/pkg/rpc/context.go#L164-L166
https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/rpc/pkg/rpc/context.go#L813-L815

cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go

Lines 91 to 93 in 046116a

	if methodExcludedFromTracing(info.FullMethod) {
	return handler(ctx, req)
	}

https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/rpc/pkg/rpc/context.go#L702-L704

cockroach/pkg/server/node.go

Lines 1718 to 1720 in 046116a

	br, err := n.batchInternal(ctx, tenantID, args)

cockroach/pkg/server/node.go

Lines 1517 to 1519 in 046116a

	var pErr *kvpb.Error
	br, writeBytes, pErr = n.stores.SendWithWriteBytes(ctx, args)
	if pErr != nil {

https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/kv/kvserver/pkg/kv/kvserver/stores.go#L196-L198
https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go#L187-L189
https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L176-L178
https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L452-L454

cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go

Lines 238 to 240 in 046116a

	g.EvalKind = evalKind
	resp, err := m.sequenceReqWithGuard(ctx, g, branch)
	if resp != nil || err != nil {

cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go

Lines 301 to 303 in 046116a

	if g.EvalKind != PessimisticAfterFailedOptimisticEval {
	panic(redact.Safe(fmt.Sprintf("must not be holding latches\n"+
	"this is tracked in github.com/cockroachdb/cockroach/issues/77663; please comment if seen\n"+

GOROOT/src/runtime/panic.go#L769-L771
https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go#L103-L105
GOROOT/src/runtime/panic.go#L769-L771

cockroach/pkg/util/stop/stopper.go

Lines 225 to 227 in 046116a

	logcrash.ReportPanicWithGlobalSettings(ctx, r, 1)
	panic(r)
	}

GOROOT/src/runtime/panic.go#L769-L771

pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges at line 1749
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch at line 2200
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas at line 2752
pkg/kv/kvclient/kvcoord/transport.go in pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext at line 187
pkg/kv/kvclient/kvcoord/transport.go in pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch at line 209
pkg/rpc/pkg/rpc/context.go in pkg/rpc.internalClientAdapter.Batch at line 906
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func3 at line 783
pkg/rpc/pkg/rpc/context.go in pkg/rpc.getChainUnaryInvoker.func1 at line 898
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/rpc.NewContext.ClientInterceptor.func8 at line 223
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func2 at line 713
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 814
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1 at line 130
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 332
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1.1 at line 132
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 814
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func3 at line 165
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 814
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/rpc.NewServerEx.ServerInterceptor.func12 at line 92
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func1 at line 703
pkg/server/node.go in pkg/server.(*Node).Batch at line 1719
pkg/server/node.go in pkg/server.(*Node).batchInternal at line 1518
pkg/kv/kvserver/pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).SendWithWriteBytes at line 197
pkg/kv/kvserver/pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes at line 188
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).SendWithWriteBytes at line 177
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries at line 453
pkg/kv/kvserver/concurrency/concurrency_manager.go in pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq at line 239
pkg/kv/kvserver/concurrency/concurrency_manager.go in pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard at line 302
GOROOT/src/runtime/panic.go in runtime.gopanic at line 770
pkg/kv/kvserver/pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes.func1 at line 104
GOROOT/src/runtime/panic.go in runtime.gopanic at line 770
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).recover at line 226
GOROOT/src/runtime/panic.go in runtime.gopanic at line 770

Tags

Tag	Value
Command	start-single-node
Environment	v24.2.10
Go Version	go1.22.5 X:nocoverageredesign
Platform	linux amd64
Distribution	CCL
Cockroach Release	v24.2.10
Cockroach SHA	046116a
# of CPUs	4
# of Goroutines	530

Jira issue: CRDB-65319

[blathers-crl]

CC'ing via the CODEOWNERS-based sentry heuristic:

• @cockroachdb/kv

Sentry issue cause: pkg/kv/kvserver/concurrency/concurrency_manager.go

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}