Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sql: v22.1.5: nil schema descriptor returned by GetImmutableSchemaByID #87895

Closed
cockroach-teamcity opened this issue Sep 13, 2022 · 0 comments · Fixed by #88568
Closed

sql: v22.1.5: nil schema descriptor returned by GetImmutableSchemaByID #87895

cockroach-teamcity opened this issue Sep 13, 2022 · 0 comments · Fixed by #88568
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Projects
SQL Foundations

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Sep 13, 2022 •
edited by cockroach-jira-scripts

This issue was autofiled by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry link: https://sentry.io/organizations/cockroach-labs/issues/3585175124/?referrer=webhooks_plugin

Panic message:

conn_executor.go:754: runtime error: invalid memory address or nil pointer dereference
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn.func1
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:754
| [...repeated from below...]
Wraps: (2) while executing: SELECT ., ., . FROM _ AS _ JOIN _ AS _ ON (. = .) JOIN _ AS _ ON (. = .) JOIN _ AS _ ON (. = .) WHERE (((. = '') AND (._ = '')) AND ( > )) AND (NOT ) ORDER BY ., .
Wraps: (3) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn.func1
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:754
| runtime.gopanic
| GOROOT/src/runtime/panic.go:1038
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:58
| runtime.gopanic
| GOROOT/src/runtime/panic.go:1038
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:58
| runtime.gopanic
| GOROOT/src/runtime/panic.go:1038
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:58
| runtime.gopanic
| GOROOT/src/runtime/panic.go:1038
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:58
| runtime.gopanic
| GOROOT/src/runtime/panic.go:1038
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:58
| runtime.gopanic
| GOROOT/src/runtime/panic.go:1038
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:58
| runtime.gopanic
| GOROOT/src/runtime/panic.go:1038
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:58
| runtime.gopanic
| GOROOT/src/runtime/panic.go:1038
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:58
| runtime.gopanic
| GOROOT/src/runtime/panic.go:1038
| runtime.panicmem
| GOROOT/src/runtime/panic.go:221
| runtime.sigpanic
| GOROOT/src/runtime/signal_windows.go:254
| github.com/cockroachdb/cockroach/pkg/sql.makeAllRelationsVirtualTableWithDescriptorIDIndex.func2
| github.com/cockroachdb/cockroach/pkg/sql/pg_catalog.go:1125
| github.com/cockroachdb/cockroach/pkg/sql.(*virtualDefEntry).makeConstrainedRowsGenerator.func1
| github.com/cockroachdb/cockroach/pkg/sql/virtual_schema.go:665
| github.com/cockroachdb/cockroach/pkg/sql.(*vTableLookupJoinNode).Next
| github.com/cockroachdb/cockroach/pkg/sql/virtual_table.go:320
| github.com/cockroachdb/cockroach/pkg/sql.(*planNodeToRowSource).Next
| github.com/cockroachdb/cockroach/pkg/sql/plan_node_to_row_source.go:191
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Columnarizer).Next
| github.com/cockroachdb/cockroach/pkg/sql/colexec/columnarizer.go:229
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).next
| github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:99
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:91
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Next
| github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:107
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexechash.(*HashTable).FullBuild
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexechash/hashtable.go:444
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecjoin.(*hashJoiner).build
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecjoin/hashjoiner.go:318
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecjoin.(*hashJoiner).Next
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecjoin/hashjoiner.go:280
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*diskSpillerBase).Next.func1
| github.com/cockroachdb/cockroach/pkg/sql/colexec/disk_spiller.go:198
Wraps: (4) runtime error: invalid memory address or nil pointer dereference
Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *withstack.withStack (4) runtime.errorString
-- report composition:
runtime.errorString
conn_executor.go:754: *withstack.withStack (top exception)
*safedetails.withSafeDetails: while executing: SELECT ., ., . FROM _ AS _ JOIN _ AS _ ON (. = .) JOIN _ AS _ ON (. = .) JOIN _ AS _ ON (. = .) WHERE (((. = '') AND (._ = '')) AND ( > _)) AND (NOT _) ORDER BY ., .
conn_executor.go:754: *withstack.withStack (1)
(check the extra data payloads)

Stacktrace (expand for inline code snippets):

cockroach/pkg/sql/conn_executor.go

Lines 753 to 755 in a30a663

r := recover()
h.ex.closeWrapper(ctx, r)
}()
in pkg/sql.(*Server).ServeConn.func1
GOROOT/src/runtime/panic.go#L1037-L1039 in runtime.gopanic

cockroach/pkg/sql/colexecerror/error.go

Lines 57 to 59 in a30a663

if !shouldCatchPanic(panicEmittedFrom) {
panic(panicObj)
}
in pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1
GOROOT/src/runtime/panic.go#L1037-L1039 in runtime.gopanic

cockroach/pkg/sql/colexecerror/error.go

Lines 57 to 59 in a30a663

if !shouldCatchPanic(panicEmittedFrom) {
panic(panicObj)
}
in pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1
GOROOT/src/runtime/panic.go#L1037-L1039 in runtime.gopanic

cockroach/pkg/sql/colexecerror/error.go

Lines 57 to 59 in a30a663

if !shouldCatchPanic(panicEmittedFrom) {
panic(panicObj)
}
in pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1
GOROOT/src/runtime/panic.go#L1037-L1039 in runtime.gopanic

cockroach/pkg/sql/colexecerror/error.go

Lines 57 to 59 in a30a663

if !shouldCatchPanic(panicEmittedFrom) {
panic(panicObj)
}
in pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1
GOROOT/src/runtime/panic.go#L1037-L1039 in runtime.gopanic

cockroach/pkg/sql/colexecerror/error.go

Lines 57 to 59 in a30a663

if !shouldCatchPanic(panicEmittedFrom) {
panic(panicObj)
}
in pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1
GOROOT/src/runtime/panic.go#L1037-L1039 in runtime.gopanic

cockroach/pkg/sql/colexecerror/error.go

Lines 57 to 59 in a30a663

if !shouldCatchPanic(panicEmittedFrom) {
panic(panicObj)
}
in pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1
GOROOT/src/runtime/panic.go#L1037-L1039 in runtime.gopanic

cockroach/pkg/sql/colexecerror/error.go

Lines 57 to 59 in a30a663

if !shouldCatchPanic(panicEmittedFrom) {
panic(panicObj)
}
in pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1
GOROOT/src/runtime/panic.go#L1037-L1039 in runtime.gopanic

cockroach/pkg/sql/colexecerror/error.go

Lines 57 to 59 in a30a663

if !shouldCatchPanic(panicEmittedFrom) {
panic(panicObj)
}
in pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1
GOROOT/src/runtime/panic.go#L1037-L1039 in runtime.gopanic
GOROOT/src/runtime/panic.go#L220-L222 in runtime.panicmem
GOROOT/src/runtime/signal_windows.go#L253-L255 in runtime.sigpanic

cockroach/pkg/sql/pg_catalog.go

Lines 1124 to 1126 in a30a663

}
if err := populateFromTable(ctx, p, h, db, sc.GetName(), table, scResolver,
addRow); err != nil {
in pkg/sql.makeAllRelationsVirtualTableWithDescriptorIDIndex.func2

cockroach/pkg/sql/virtual_schema.go

Lines 664 to 666 in a30a663

// constraint span's value.
found, err := virtualIndex.populate(ctx, constraintDatum, p, dbDesc,
addRowIfPassesFilter(idxConstraint))
in pkg/sql.(*virtualDefEntry).makeConstrainedRowsGenerator.func1

cockroach/pkg/sql/virtual_table.go

Lines 319 to 321 in a30a663

// the looked-up rows into v.run.rows.
if err := genFunc(params.ctx, v); err != nil {
return false, err
in pkg/sql.(*vTableLookupJoinNode).Next

cockroach/pkg/sql/plan_node_to_row_source.go

Lines 190 to 192 in a30a663

for p.State == execinfra.StateRunning {
valid, err := p.node.Next(p.params)
if err != nil || !valid {
in pkg/sql.(*planNodeToRowSource).Next

cockroach/pkg/sql/colexec/columnarizer.go

Lines 228 to 230 in a30a663

for ; nRows < c.batch.Capacity(); nRows++ {
row, meta := c.input.Next()
if meta != nil {
in pkg/sql/colexec.(*Columnarizer).Next

cockroach/pkg/sql/colflow/stats.go

Lines 98 to 100 in a30a663

func (bic *batchInfoCollector) next() {
bic.batch = bic.Operator.Next()
}
in pkg/sql/colflow.(*batchInfoCollector).next

cockroach/pkg/sql/colexecerror/error.go

Lines 90 to 92 in a30a663

}()
operation()
return retErr
in pkg/sql/colexecerror.CatchVectorizedRuntimeError

cockroach/pkg/sql/colflow/stats.go

Lines 106 to 108 in a30a663

// execution time (e.g. in the statement bundle).
err := colexecerror.CatchVectorizedRuntimeError(bic.next)
bic.stopwatch.Stop()
in pkg/sql/colflow.(*batchInfoCollector).Next

cockroach/pkg/sql/colexec/colexechash/hashtable.go

Lines 443 to 445 in a30a663

for {
batch := input.Next()
if batch.Length() == 0 {
in pkg/sql/colexec/colexechash.(*HashTable).FullBuild

cockroach/pkg/sql/colexec/colexecjoin/hashjoiner.go

Lines 317 to 319 in a30a663

func (hj *hashJoiner) build() {
hj.ht.FullBuild(hj.inputTwo)
in pkg/sql/colexec/colexecjoin.(*hashJoiner).build

cockroach/pkg/sql/colexec/colexecjoin/hashjoiner.go

Lines 279 to 281 in a30a663

case hjBuilding:
hj.build()
if hj.ht.Vals.Length() == 0 {
in pkg/sql/colexec/colexecjoin.(*hashJoiner).Next

cockroach/pkg/sql/colexec/disk_spiller.go

Lines 197 to 199 in a30a663

func() {
batch = d.inMemoryOp.Next()
},
in pkg/sql/colexec.(*diskSpillerBase).Next.func1

pkg/sql/conn_executor.go in pkg/sql.(*Server).ServeConn.func1 at line 754
GOROOT/src/runtime/panic.go in runtime.gopanic at line 1038
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1 at line 58
GOROOT/src/runtime/panic.go in runtime.gopanic at line 1038
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1 at line 58
GOROOT/src/runtime/panic.go in runtime.gopanic at line 1038
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1 at line 58
GOROOT/src/runtime/panic.go in runtime.gopanic at line 1038
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1 at line 58
GOROOT/src/runtime/panic.go in runtime.gopanic at line 1038
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1 at line 58
GOROOT/src/runtime/panic.go in runtime.gopanic at line 1038
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1 at line 58
GOROOT/src/runtime/panic.go in runtime.gopanic at line 1038
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1 at line 58
GOROOT/src/runtime/panic.go in runtime.gopanic at line 1038
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1 at line 58
GOROOT/src/runtime/panic.go in runtime.gopanic at line 1038
GOROOT/src/runtime/panic.go in runtime.panicmem at line 221
GOROOT/src/runtime/signal_windows.go in runtime.sigpanic at line 254
pkg/sql/pg_catalog.go in pkg/sql.makeAllRelationsVirtualTableWithDescriptorIDIndex.func2 at line 1125
pkg/sql/virtual_schema.go in pkg/sql.(*virtualDefEntry).makeConstrainedRowsGenerator.func1 at line 665
pkg/sql/virtual_table.go in pkg/sql.(*vTableLookupJoinNode).Next at line 320
pkg/sql/plan_node_to_row_source.go in pkg/sql.(*planNodeToRowSource).Next at line 191
pkg/sql/colexec/columnarizer.go in pkg/sql/colexec.(*Columnarizer).Next at line 229
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).next at line 99
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 91
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).Next at line 107
pkg/sql/colexec/colexechash/hashtable.go in pkg/sql/colexec/colexechash.(*HashTable).FullBuild at line 444
pkg/sql/colexec/colexecjoin/hashjoiner.go in pkg/sql/colexec/colexecjoin.(*hashJoiner).build at line 318
pkg/sql/colexec/colexecjoin/hashjoiner.go in pkg/sql/colexec/colexecjoin.(*hashJoiner).Next at line 280
pkg/sql/colexec/disk_spiller.go in pkg/sql/colexec.(*diskSpillerBase).Next.func1 at line 198
Tag Value
Cockroach Release v22.1.5
Cockroach SHA: a30a663
Platform windows amd64
Distribution CCL
Environment v22.1.5
Command start-single-node
Go Version ``
# of CPUs
# of Goroutines

Jira issue: CRDB-19581
@cockroach-teamcity cockroach-teamcity added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. labels Sep 13, 2022
@yuzefovich yuzefovich changed the title sentry: conn_executor.go:754: runtime error: invalid memory address or nil pointer dereference (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn.func1... sql: v22.1.5: nil schema descriptor returned by GetImmutableSchemaByID Sep 21, 2022
@yuzefovich yuzefovich added this to Triage in SQL Foundations via automation Sep 21, 2022
@blathers-crl blathers-crl bot added the T-sql-schema-deprecated Use T-sql-foundations instead label Sep 21, 2022
ajwerner added a commit to ajwerner/cockroach that referenced this issue Sep 23, 2022
@ajwerner
sql: fix panic due to missing schema
143c5f8 
A schema might not exist because it has been dropped. We need to mark the
lookup as required.

Fixes cockroachdb#87895

Release note (bug fix): Fixed a bug in pg_catalog tables which could result in
an internal error if a schema is concurrently dropped.
@ajwerner ajwerner mentioned this issue Sep 23, 2022
Merged
craig bot pushed a commit that referenced this issue Sep 23, 2022
@dhartunian @aadityasondhi
@adityamaru @ajwerner @stevendanna @pav-kv
Merge #87533 #88293 #88471 #88557 #88562 #88568
2c18ed3 
87533: sqlliveness: add timeouts to heartbeats r=ajwerner a=aadityasondhi

Previously, sqlliveness heartbeat operations could block on the transactions that were involved. This change introduces some timeouts of the length of the heartbeat during the create and refresh operations.

Resolves #85541

Release note: None

Release justification: low-risk bugfix to existing functionality

88293: backupccl: elide expensive ShowCreate call in SHOW BACKUP r=stevendanna a=adityamaru

In #88376 we see the call to `ShowCreate` taking ~all the time on a cluster with
2.5K empty tables. In all cases except `SHOW BACKUP SCHEMAS` we do not
need to construct the SQL representation of the table's schema. This
results in a marked improvement in the performance of `SHOW BACKUP`
as can be seen in #88376 (comment).

Fixes: #88376

Release note (performance improvement): `SHOW BACKUP` on a backup containing
several table descriptors is now more performant

88471: sql/schemachanger: plumb context, check for cancelation sometimes r=ajwerner a=ajwerner

Fixes #87246

This will also improve tracing.

Release note: None

88557: testserver: add ShareMostTestingKnobsWithTenant option r=msbutler a=stevendanna

The new ShareMostTestingKnobs copies nearly all of the testing knobs specified for a TestServer to any tenant started for that server.

The goal here is to make it easier to write tests that depend on testing hooks that work under probabilistic tenant testing.

Release justification: non-production code change

Release note: None

88562: upgrade grpc to v.1.49.0 r=erikgrinaker a=pavelkalinnikov

Fixes #81881
Touches #72083

Release note: upgraded grpc to v1.49.0 to fix a few panics that the old version caused

88568: sql: fix panic due to missing schema r=ajwerner a=ajwerner

A schema might not exist because it has been dropped. We need to mark the lookup as required.

Fixes #87895

Release note (bug fix): Fixed a bug in pg_catalog tables which could result in an internal error if a schema is concurrently dropped.

Co-authored-by: David Hartunian <davidh@cockroachlabs.com>
Co-authored-by: Aaditya Sondhi <aadityas@cockroachlabs.com>
Co-authored-by: adityamaru <adityamaru@gmail.com>
Co-authored-by: Andrew Werner <awerner32@gmail.com>
Co-authored-by: Steven Danna <danna@cockroachlabs.com>
Co-authored-by: Pavel Kalinnikov <pavel@cockroachlabs.com>
@craig craig bot closed this as completed in 5ae96bd Sep 23, 2022
SQL Foundations automation moved this from Triage to Closed Sep 23, 2022
@craig craig bot closed this as completed in #88568 Sep 23, 2022
@blathers-crl blathers-crl bot mentioned this issue Sep 23, 2022
Merged
blathers-crl bot pushed a commit that referenced this issue Sep 23, 2022
@ajwerner
sql: fix panic due to missing schema
837445b 
A schema might not exist because it has been dropped. We need to mark the
lookup as required.

Fixes #87895

Release note (bug fix): Fixed a bug in pg_catalog tables which could result in
an internal error if a schema is concurrently dropped.
blathers-crl bot pushed a commit that referenced this issue Sep 23, 2022
@ajwerner
sql: fix panic due to missing schema
77033f3 
A schema might not exist because it has been dropped. We need to mark the
lookup as required.

Fixes #87895

Release note (bug fix): Fixed a bug in pg_catalog tables which could result in
an internal error if a schema is concurrently dropped.
@blathers-crl blathers-crl bot mentioned this issue Sep 23, 2022
Merged
@yuzefovich yuzefovich mentioned this issue Oct 29, 2022
Closed
@yuzefovich yuzefovich mentioned this issue Apr 4, 2023
Closed
@exalate-issue-sync exalate-issue-sync bot reopened this May 10, 2023
@exalate-issue-sync exalate-issue-sync bot added T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) and removed T-sql-schema-deprecated Use T-sql-foundations instead labels May 10, 2023
@blathers-crl blathers-crl bot added this to Triage in SQL Foundations May 10, 2023
@rafiss rafiss closed this as completed May 10, 2023
SQL Foundations automation moved this from Triage [old] to Done May 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Projects
SQL Foundations
  
Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

sql: fix panic due to missing schema ajwerner/cockroach
2 participants
@rafiss @cockroach-teamcity