panic: default zone config missing unexpectedly [now has a reliable repro] #43951
Labels
branch-release-23.2
Used to mark GA and release blockers and technical advisories for 23.2
C-test-failure
Broken test (automatically or manually discovered).
O-rsg
Random Syntax Generator
P-3
Issues/test failures with no fix SLA
T-sql-foundations
SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Projects
Comments
maddyblue
added
C-bug
|
ping @andreimatei |
|
I also am able to hit this stressing those random syntax tests. |
yuzefovich
added
C-test-failure
|
Recent repro on 22.2.17 branch #114879. Perhaps there is a race between default zone config being set and reports running? |
kvoli
added
the
branch-release-22.2
|
No repro on 9efa484 over 10mins. |
rafiss
added
branch-release-23.2
|
Happened on 23.2 here: #117081 I found a reliable way to repro this: |
rafiss
changed the title
|
Thanks for the reproduction @rafiss, it's working for me as well. In it, we see that the diff --git a/pkg/kv/kvserver/reports/reporter.go b/pkg/kv/kvserver/reports/reporter.go
index 40346701b3a..5853cd4516e 100644
--- a/pkg/kv/kvserver/reports/reporter.go
+++ b/pkg/kv/kvserver/reports/reporter.go
@@ -196,6 +196,13 @@ func (stats *Reporter) update(
if stats.latestConfig == nil {
return nil
}
+ zone, err := getZoneByID(keys.RootNamespaceID, stats.latestConfig)
+ if err != nil {
+ log.Fatalf(ctx, "failed to get default zone config: %s", err)
+ }
+ if zone == nil {
+ log.Fatalf(ctx, "default zone config missing unexpectedly (%T): %+v", stats.cfgs, stats.latestConfig)
+ }
allStores := stats.storePool.GetStores()
var storesFromGossip StoreResolver = func(Using that diff, we also see the Instrumenting
I'll then hand the remainder of this investigation back to SQL Foundations. Specifically, we still need to determine why the row in |
nvanbenschoten
added a commit
to nvanbenschoten/cockroach
that referenced
this issue
Jan 11, 2024
Informs cockroachdb#43951. In cockroachdb#43951, we saw that some random sequence on queries led to the default zone configuration in the system.zones table (key `/Table/5/1/0/2/1`) being deleted. This led to a fatal log in the replication report generation code. Deleting this key should not be allowed, but it also shouldn't crash cockroach. This commit replaces the fatal log with an error, which is logged a few stack frames up with a gentle "some reports have not been generated" message. I've confirmed that with this fix, the reproduction steps in cockroachdb#43951 no longer lead to a fatal log and a crashed process. Release note: None
blathers-crl
bot
added
the
T-sql-foundations
nvanbenschoten
removed
T-sql-foundations
craig bot
pushed a commit
that referenced
this issue
Jan 11, 2024
117662: kv/reports: don't panic on missing default zone config r=nvanbenschoten a=nvanbenschoten Informs #43951. In #43951, we saw that some random sequence on queries led to the default zone configuration in the system.zones table (key `/Table/5/1/0/2/1`) being deleted. This led to a fatal log in the replication report generation code. Deleting this key should not be allowed, but it also shouldn't crash cockroach. This commit replaces the fatal log with an error, which is logged a few stack frames up with a gentle "some reports have not been generated" message. I've confirmed that with this fix, the reproduction steps in #43951 no longer lead to a fatal log and a crashed process. Release note: None 117663: ci: migrate CI docker images to GCP r=jlinder a=rail Previously, we used Docker Hub to host our CI docker images. To improve service reliability, this PR moves the used docker images to GAR. Part of: DEVINF-915 Epic: RE-539 Release note: None Co-authored-by: Nathan VanBenschoten <nvanbenschoten@gmail.com> Co-authored-by: Rail Aliiev <rail@iqchoice.com>
blathers-crl
bot
added
the
A-storage
nicktrav
added
sync-me
and removed
A-storage
nvanbenschoten
changed the title
exalate-issue-sync
bot
added
T-sql-foundations
|
We have marked this test failure issue as stale because it has been |
andrewbaptist
pushed a commit
to andrewbaptist/cockroach
that referenced
this issue
Mar 6, 2024
Informs cockroachdb#43951. In cockroachdb#43951, we saw that some random sequence on queries led to the default zone configuration in the system.zones table (key `/Table/5/1/0/2/1`) being deleted. This led to a fatal log in the replication report generation code. Deleting this key should not be allowed, but it also shouldn't crash cockroach. This commit replaces the fatal log with an error, which is logged a few stack frames up with a gentle "some reports have not been generated" message. I've confirmed that with this fix, the reproduction steps in cockroachdb#43951 no longer lead to a fatal log and a crashed process. Release note: None
|
We have marked this test failure issue as stale because it has been |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The RSG tests appear to have done a bad thing: https://teamcity.cockroachdb.com/viewLog.html?buildId=1688020&buildTypeId=Cockroach_Nightlies_RandomSyntaxTests
Full stack trace and logs at https://teamcity.cockroachdb.com/downloadBuildLog.html?buildId=1688020&plain=true
Jira issue: CRDB-5266
The text was updated successfully, but these errors were encountered: