v22.1.0-beta.3: kvserver: panic with default zone config missing unexpectedly

[cockroach-teamcity]

This issue was autofiled by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry link: https://sentry.io/organizations/cockroach-labs/issues/3214525872/?referrer=webhooks_plugin

Panic message:

reporter.go:493: log.Fatal: default zone config missing unexpectedly
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/reports.visitDefaultZone
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/reports/reporter.go:493
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/reports.visitAncestors
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/reports/reporter.go:454
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/reports.visitZones
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/reports/reporter.go:437
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/reports.(*zoneResolver).updateZone
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/reports/reporter.go:341
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/reports.(*zoneResolver).resolveRange
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/reports/reporter.go:313
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/reports.visitRanges
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/reports/reporter.go:598
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/reports.(*Reporter).update
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/reports/reporter.go:231
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/reports.(*Reporter).Start.func2
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/reports/reporter.go:150
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:494
| runtime.goexit
| GOROOT/src/runtime/asm_amd64.s:1581
Wraps: (2) log.Fatal: default zone config missing unexpectedly
Error types: (1) *withstack.withStack (2) *errutil.leafError
-- report composition:
*errutil.leafError: log.Fatal: default zone config missing unexpectedly
reporter.go:493: *withstack.withStack (top exception)

Stacktrace (expand for inline code snippets):

cockroach/pkg/kv/kvserver/reports/reporter.go

Lines 492 to 494 in 0509bc6

	if zone == nil {
	log.Fatal(ctx, "default zone config missing unexpectedly")
	}

in pkg/kv/kvserver/reports.visitDefaultZone

cockroach/pkg/kv/kvserver/reports/reporter.go

Lines 453 to 455 in 0509bc6

	// Let's just look at the default zone config.
	return visitDefaultZone(ctx, cfg, visitor), nil
	}

in pkg/kv/kvserver/reports.visitAncestors

cockroach/pkg/kv/kvserver/reports/reporter.go

Lines 436 to 438 in 0509bc6

	// Go upwards.
	return visitAncestors(ctx, id, cfg, visitor)
	}

in pkg/kv/kvserver/reports.visitZones

cockroach/pkg/kv/kvserver/reports/reporter.go

Lines 340 to 342 in 0509bc6

	// visitZones() walks the zone hierarchy from the bottom upwards.
	found, err := visitZones(
	ctx, rd, cfg, includeSubzonePlaceholders,

in pkg/kv/kvserver/reports.(*zoneResolver).updateZone

cockroach/pkg/kv/kvserver/reports/reporter.go

Lines 312 to 314 in 0509bc6

	}
	return c.updateZone(ctx, rng, cfg)
	}

in pkg/kv/kvserver/reports.(*zoneResolver).resolveRange

cockroach/pkg/kv/kvserver/reports/reporter.go

Lines 597 to 599 in 0509bc6

	newKey, err := resolver.resolveRange(ctx, &rd, cfg)
	if err != nil {

in pkg/kv/kvserver/reports.visitRanges

cockroach/pkg/kv/kvserver/reports/reporter.go

Lines 230 to 232 in 0509bc6

	rangeIter := makeMeta2RangeIter(stats.db, descriptorReadBatchSize)
	if err := visitRanges(
	ctx, &rangeIter, stats.latestConfig,

in pkg/kv/kvserver/reports.(*Reporter).update

cockroach/pkg/kv/kvserver/reports/reporter.go

Lines 149 to 151 in 0509bc6

	if stats.meta1LeaseHolder != nil {
	if err := stats.update(
	ctx, &constraintsSaver, &replStatsSaver, &criticalLocSaver,

in pkg/kv/kvserver/reports.(*Reporter).Start.func2

cockroach/pkg/util/stop/stopper.go

Lines 493 to 495 in 0509bc6

	f(ctx)
	}()

in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
GOROOT/src/runtime/asm_amd64.s#L1580-L1582 in runtime.goexit

pkg/kv/kvserver/reports/reporter.go in pkg/kv/kvserver/reports.visitDefaultZone at line 493
pkg/kv/kvserver/reports/reporter.go in pkg/kv/kvserver/reports.visitAncestors at line 454
pkg/kv/kvserver/reports/reporter.go in pkg/kv/kvserver/reports.visitZones at line 437
pkg/kv/kvserver/reports/reporter.go in pkg/kv/kvserver/reports.(*zoneResolver).updateZone at line 341
pkg/kv/kvserver/reports/reporter.go in pkg/kv/kvserver/reports.(*zoneResolver).resolveRange at line 313
pkg/kv/kvserver/reports/reporter.go in pkg/kv/kvserver/reports.visitRanges at line 598
pkg/kv/kvserver/reports/reporter.go in pkg/kv/kvserver/reports.(*Reporter).update at line 231
pkg/kv/kvserver/reports/reporter.go in pkg/kv/kvserver/reports.(*Reporter).Start.func2 at line 150
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 494
GOROOT/src/runtime/asm_amd64.s in runtime.goexit at line 1581

Tag	Value
Cockroach Release	v22.1.0-beta.3
Cockroach SHA:	0509bc6
Platform	linux amd64
Distribution	CCL
Environment	v22.1.0-beta.3
Command	demo
Go Version	``
# of CPUs
# of Goroutines

Jira issue: CRDB-15764

[rafiss]

This looks like #43951 except it happened out in the wild.

cc @irfansharif ?

[irfansharif]

Unsure if it's due to the recent changes in the area. The area where the panic occurred didn't directly change this past release, except perhaps as part of @ajwerner's #74612. I wonder if it in fact is the same as #43951.

[mwang1026]

stale