Sentry: panic.go:884: runtime error: invalid memory address or nil pointer dereference (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/panic.go:884 | runtime.p...

[cockroach-sentry]

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/5355001773/?referrer=webhooks_plugin

Panic Message:

panic.go:884: runtime error: invalid memory address or nil pointer dereference
(1) attached stack trace
  -- stack trace:
  | runtime.gopanic
  | 	GOROOT/src/runtime/panic.go:884
  | runtime.panicmem
  | 	GOROOT/src/runtime/panic.go:260
  | runtime.sigpanic
  | 	GOROOT/src/runtime/signal_unix.go:839
  | google.golang.org/grpc/internal/metadata.Validate
  | 	google.golang.org/grpc/internal/metadata/external/org_golang_google_grpc/internal/metadata/metadata.go:86
  | google.golang.org/grpc.newClientStream
  | 	google.golang.org/grpc/external/org_golang_google_grpc/stream.go:172
  | google.golang.org/grpc.invoke
  | 	google.golang.org/grpc/external/org_golang_google_grpc/call.go:66
  | github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor.ClientInterceptor.func2
  | 	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:248
  | google.golang.org/grpc.(*ClientConn).Invoke
  | 	google.golang.org/grpc/external/org_golang_google_grpc/call.go:35
  | github.com/cockroachdb/cockroach/pkg/rpc.(*heartbeatClient).Ping
  | 	github.com/cockroachdb/cockroach/pkg/rpc/bazel-out/k8-opt/bin/pkg/rpc/rpc_go_proto_/github.com/cockroachdb/cockroach/pkg/rpc/heartbeat.pb.go:317
  | github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat.func2.2
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:2504
  | github.com/cockroachdb/cockroach/pkg/util/contextutil.RunWithTimeout
  | 	github.com/cockroachdb/cockroach/pkg/util/contextutil/context.go:91
  | github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat.func2
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:2509
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
  | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:322
  | github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:2476
  | github.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:2317
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
  | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470
  | runtime.goexit
  | 	GOROOT/src/runtime/asm_amd64.s:1594
Wraps: (2) runtime error: invalid memory address or nil pointer dereference
Error types: (1) *withstack.withStack (2) runtime.errorString
-- report composition:
runtime.errorString
panic.go:884: *withstack.withStack (top exception)

Stacktrace (expand for inline code snippets):

GOROOT/src/runtime/asm_amd64.s#L1593-L1595

cockroach/pkg/util/stop/stopper.go

Lines 469 to 471 in 28b2e76

	sp.UpdateGoroutineIDToCurrent()
	f(ctx)
	}()

https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/rpc/pkg/rpc/context.go#L2316-L2318
https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/rpc/pkg/rpc/context.go#L2475-L2477

cockroach/pkg/util/stop/stopper.go

Lines 321 to 323 in 28b2e76

	return f(ctx)
	}

https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/rpc/pkg/rpc/context.go#L2508-L2510

cockroach/pkg/util/contextutil/context.go

Lines 90 to 92 in 28b2e76

	start := timeutil.Now()
	err := fn(ctx)
	if err != nil && errors.Is(ctx.Err(), context.DeadlineExceeded) {

https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/rpc/pkg/rpc/context.go#L2503-L2505
https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/rpc/bazel-out/k8-opt/bin/pkg/rpc/rpc_go_proto_/github.com/cockroachdb/cockroach/pkg/rpc/heartbeat.pb.go#L316-L318
google.golang.org/grpc/external/org_golang_google_grpc/call.go#L34-L36

cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go

Lines 247 to 249 in 28b2e76

	if invoker != nil {
	err := invoker(ctx, method, req, resp, cc, opts...)
	if err != nil {

google.golang.org/grpc/external/org_golang_google_grpc/call.go#L65-L67
google.golang.org/grpc/external/org_golang_google_grpc/stream.go#L171-L173
google.golang.org/grpc/internal/metadata/external/org_golang_google_grpc/internal/metadata/metadata.go#L85-L87
GOROOT/src/runtime/signal_unix.go#L838-L840
GOROOT/src/runtime/panic.go#L259-L261
GOROOT/src/runtime/panic.go#L883-L885

GOROOT/src/runtime/asm_amd64.s in runtime.goexit at line 1594
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 470
pkg/rpc/pkg/rpc/context.go in pkg/rpc.(*Context).grpcDialNodeInternal.func1 at line 2317
pkg/rpc/pkg/rpc/context.go in pkg/rpc.(*Context).runHeartbeat at line 2476
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 322
pkg/rpc/pkg/rpc/context.go in pkg/rpc.(*Context).runHeartbeat.func2 at line 2509
pkg/util/contextutil/context.go in pkg/util/contextutil.RunWithTimeout at line 91
pkg/rpc/pkg/rpc/context.go in pkg/rpc.(*Context).runHeartbeat.func2.2 at line 2504
pkg/rpc/bazel-out/k8-opt/bin/pkg/rpc/rpc_go_proto_/github.com/cockroachdb/cockroach/pkg/rpc/heartbeat.pb.go in pkg/rpc.(*heartbeatClient).Ping at line 317
google.golang.org/grpc/external/org_golang_google_grpc/call.go in google.golang.org/grpc.(*ClientConn).Invoke at line 35
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/util/tracing/grpcinterceptor.ClientInterceptor.func2 at line 248
google.golang.org/grpc/external/org_golang_google_grpc/call.go in google.golang.org/grpc.invoke at line 66
google.golang.org/grpc/external/org_golang_google_grpc/stream.go in google.golang.org/grpc.newClientStream at line 172
google.golang.org/grpc/internal/metadata/external/org_golang_google_grpc/internal/metadata/metadata.go in google.golang.org/grpc/internal/metadata.Validate at line 86
GOROOT/src/runtime/signal_unix.go in runtime.sigpanic at line 839
GOROOT/src/runtime/panic.go in runtime.panicmem at line 260
GOROOT/src/runtime/panic.go in runtime.gopanic at line 884

Tags

Tag	Value
Command	server
Environment	v23.1.9
Go Version	go1.19.10
Platform	linux amd64
Distribution	CCL
Cockroach Release	v23.1.9
Cockroach SHA	28b2e76
# of CPUs	8
# of Goroutines	454

Jira issue: CRDB-38784

[yuzefovich]

unactionable - seems like a bug in the upstream library