backfill: v22.2.0: nil txn crash in cluster_logical_timestamp builtin in backfill #93870

cockroach-teamcity · 2022-12-19T00:14:52Z

This issue was autofiled by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry link: https://sentry.io/organizations/cockroach-labs/issues/3821279521/?referrer=webhooks_plugin

Panic message:

panic.go:884: runtime error: invalid memory address or nil pointer dereference
(1) attached stack trace
-- stack trace:
| runtime.gopanic
| GOROOT/src/runtime/panic.go:884
| runtime.panicmem
| GOROOT/src/runtime/panic.go:260
| runtime.sigpanic
| GOROOT/src/runtime/signal_unix.go:835
| github.com/cockroachdb/cockroach/pkg/kv.(*Txn).CommitTimestamp
| github.com/cockroachdb/cockroach/pkg/kv/txn.go:373
| github.com/cockroachdb/cockroach/pkg/sql/sem/eval.(*Context).GetClusterTimestamp
| github.com/cockroachdb/cockroach/pkg/sql/sem/eval/context.go:406
| github.com/cockroachdb/cockroach/pkg/sql/sem/builtins.glob..func148
| github.com/cockroachdb/cockroach/pkg/sql/sem/builtins/builtins.go:2785
| github.com/cockroachdb/cockroach/pkg/sql/sem/eval.(*evaluator).EvalFuncExpr
| github.com/cockroachdb/cockroach/pkg/sql/sem/eval/expr.go:459
| github.com/cockroachdb/cockroach/pkg/sql/sem/tree.(*FuncExpr).Eval
| github.com/cockroachdb/cockroach/bazel-out/k8-opt/bin/pkg/sql/sem/tree/eval_expr_generated.go:264
| github.com/cockroachdb/cockroach/pkg/sql/sem/eval.Expr
| github.com/cockroachdb/cockroach/pkg/sql/sem/eval/expr.go:24
| github.com/cockroachdb/cockroach/pkg/sql/backfill.(*ColumnBackfiller).RunColumnBackfillChunk
| github.com/cockroachdb/cockroach/pkg/sql/backfill/backfill.go:370
| github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*columnBackfiller).runChunk.func1
| github.com/cockroachdb/cockroach/pkg/sql/rowexec/columnbackfiller.go:132
| github.com/cockroachdb/cockroach/pkg/kv.runTxn.func1
| github.com/cockroachdb/cockroach/pkg/kv/db.go:975
| github.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec
| github.com/cockroachdb/cockroach/pkg/kv/txn.go:961
| github.com/cockroachdb/cockroach/pkg/kv.runTxn
| github.com/cockroachdb/cockroach/pkg/kv/db.go:974
| github.com/cockroachdb/cockroach/pkg/kv.(*DB).TxnWithAdmissionControl
| github.com/cockroachdb/cockroach/pkg/kv/db.go:937
| github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*columnBackfiller).runChunk
| github.com/cockroachdb/cockroach/pkg/sql/rowexec/columnbackfiller.go:114
| github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*backfiller).mainLoop
| github.com/cockroachdb/cockroach/pkg/sql/rowexec/backfiller.go:154
| github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*backfiller).doRun
| github.com/cockroachdb/cockroach/pkg/sql/rowexec/backfiller.go:107
| github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*backfiller).Run
| github.com/cockroachdb/cockroach/pkg/sql/rowexec/backfiller.go:95
| github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).Run
| github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow.go:472
| github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run
| github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:695
| github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).distColumnBackfill.func3
| github.com/cockroachdb/cockroach/pkg/sql/backfill.go:1348
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*CollectionFactory).Txn.func1
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/txn.go:52
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*CollectionFactory).TxnWithExecutor.func3
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/txn.go:163
| github.com/cockroachdb/cockroach/pkg/kv.runTxn.func1
| github.com/cockroachdb/cockroach/pkg/kv/db.go:975
| github.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec
| github.com/cockroachdb/cockroach/pkg/kv/txn.go:961
| github.com/cockroachdb/cockroach/pkg/kv.runTxn
| github.com/cockroachdb/cockroach/pkg/kv/db.go:974
| github.com/cockroachdb/cockroach/pkg/kv.(*DB).TxnWithAdmissionControl
| github.com/cockroachdb/cockroach/pkg/kv/db.go:937
| github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn
| github.com/cockroachdb/cockroach/pkg/kv/db.go:916
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*CollectionFactory).TxnWithExecutor
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/txn.go:155
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*CollectionFactory).Txn
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/txn.go:49
| github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).txn
| github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:2464
Wraps: (2) runtime error: invalid memory address or nil pointer dereference
Error types: (1) *withstack.withStack (2) runtime.errorString
-- report composition:
runtime.errorString
panic.go:884: *withstack.withStack (top exception)

Stacktrace (expand for inline code snippets):

GOROOT/src/runtime/panic.go#L883-L885 in runtime.gopanic
GOROOT/src/runtime/panic.go#L259-L261 in runtime.panicmem
GOROOT/src/runtime/signal_unix.go#L834-L836 in runtime.sigpanic

cockroach/pkg/kv/txn.go

Lines 372 to 374 in 77667a1

    
           func (txn *Txn) CommitTimestamp() hlc.Timestamp { 
        
           	txn.mu.Lock() 
        
           	defer txn.mu.Unlock()

in pkg/kv.(*Txn).CommitTimestamp

cockroach/pkg/sql/sem/eval/context.go

Lines 405 to 407 in 77667a1

    
           func (ec *Context) GetClusterTimestamp() *tree.DDecimal { 
        
           	ts := ec.Txn.CommitTimestamp() 
        
           	if ts.IsEmpty() {

in pkg/sql/sem/eval.(*Context).GetClusterTimestamp

cockroach/pkg/sql/sem/builtins/builtins.go

Lines 2784 to 2786 in 77667a1

    
           Fn: func(ctx *eval.Context, args tree.Datums) (tree.Datum, error) { 
        
           	return ctx.GetClusterTimestamp(), nil 
        
           },

in pkg/sql/sem/builtins.glob..func148

cockroach/pkg/sql/sem/eval/expr.go

Lines 458 to 460 in 77667a1

    
           res, err := fn.Fn.(FnOverload)(e.ctx(), args) 
        
           if err != nil {

in pkg/sql/sem/eval.(*evaluator).EvalFuncExpr
https://github.com/cockroachdb/cockroach/blob/77667a1b0101cd323090011f50cf910aaa933654/bazel-out/k8-opt/bin/pkg/sql/sem/tree/eval_expr_generated.go#L263-L265 in pkg/sql/sem/tree.(*FuncExpr).Eval

cockroach/pkg/sql/sem/eval/expr.go

Lines 23 to 25 in 77667a1

    
           func Expr(ctx *Context, n tree.TypedExpr) (tree.Datum, error) { 
        
           	return n.Eval((*evaluator)(ctx)) 
        
           }

in pkg/sql/sem/eval.Expr

cockroach/pkg/sql/backfill/backfill.go

Lines 369 to 371 in 77667a1

    
           for j, e := range cb.updateExprs { 
        
           	val, err := eval.Expr(cb.evalCtx, e) 
        
           	if err != nil {

in pkg/sql/backfill.(*ColumnBackfiller).RunColumnBackfillChunk

cockroach/pkg/sql/rowexec/columnbackfiller.go

Lines 131 to 133 in 77667a1

    
           var err error 
        
           key, err = cb.RunColumnBackfillChunk( 
        
           	ctx,

in pkg/sql/rowexec.(*columnBackfiller).runChunk.func1

cockroach/pkg/kv/db.go

Lines 974 to 976 in 77667a1

    
           err := txn.exec(ctx, func(ctx context.Context, txn *Txn) error { 
        
           	return retryable(ctx, txn) 
        
           })

in pkg/kv.runTxn.func1

cockroach/pkg/kv/txn.go

Lines 960 to 962 in 77667a1

    
           } 
        
           err = fn(ctx, txn)

in pkg/kv.(*Txn).exec

cockroach/pkg/kv/db.go

Lines 973 to 975 in 77667a1

    
           func runTxn(ctx context.Context, txn *Txn, retryable func(context.Context, *Txn) error) error { 
        
           	err := txn.exec(ctx, func(ctx context.Context, txn *Txn) error { 
        
           		return retryable(ctx, txn)

in pkg/kv.runTxn

cockroach/pkg/kv/db.go

Lines 936 to 938 in 77667a1

    
           	txn.SetDebugName("unnamed") 
        
           	return runTxn(ctx, txn, retryable) 
        
           }

in pkg/kv.(*DB).TxnWithAdmissionControl

cockroach/pkg/sql/rowexec/columnbackfiller.go

Lines 113 to 115 in 77667a1

    
           var commitWaitFn func(context.Context) error 
        
           err := cb.flowCtx.Cfg.DB.TxnWithAdmissionControl( 
        
           	ctx, roachpb.AdmissionHeader_FROM_SQL, admissionpb.BulkNormalPri,

in pkg/sql/rowexec.(*columnBackfiller).runChunk

cockroach/pkg/sql/rowexec/backfiller.go

Lines 153 to 155 in 77667a1

    
           var err error 
        
           todo.Key, err = b.chunks.runChunk(ctx, todo, chunkSize, updateChunkSizeThresholdBytes, b.spec.ReadAsOf) 
        
           if err != nil {

in pkg/sql/rowexec.(*backfiller).mainLoop

cockroach/pkg/sql/rowexec/backfiller.go

Lines 106 to 108 in 77667a1

    
           } 
        
           finishedSpans, err := b.mainLoop(ctx) 
        
           if err != nil {

in pkg/sql/rowexec.(*backfiller).doRun

cockroach/pkg/sql/rowexec/backfiller.go

Lines 94 to 96 in 77667a1

    
           defer span.Finish() 
        
           meta := b.doRun(ctx) 
        
           execinfra.SendTraceData(ctx, b.output)

in pkg/sql/rowexec.(*backfiller).Run

cockroach/pkg/sql/flowinfra/flow.go

Lines 471 to 473 in 77667a1

    
           	log.VEventf(ctx, 1, "running %T in the flow's goroutine", headProc) 
        
           	headProc.Run(ctx) 
        
           }

in pkg/sql/flowinfra.(*FlowBase).Run

cockroach/pkg/sql/distsql_running.go

Lines 694 to 696 in 77667a1

    
           // TODO(radu): this should go through the flow scheduler. 
        
           flow.Run(ctx, func() {})

in pkg/sql.(*DistSQLPlanner).Run

cockroach/pkg/sql/backfill.go

Lines 1347 to 1349 in 77667a1

    
           } 
        
           sc.distSQLPlanner.Run( 
        
           	ctx,

in pkg/sql.(*SchemaChanger).distColumnBackfill.func3

cockroach/pkg/sql/catalog/descs/txn.go

Lines 51 to 53 in 77667a1

    
           ) error { 
        
           	return f(ctx, txn, descriptors) 
        
           }, opts...)

in pkg/sql/catalog/descs.(*CollectionFactory).Txn.func1

cockroach/pkg/sql/catalog/descs/txn.go

Lines 162 to 164 in 77667a1

    
           ie, commitTxnFn := cf.ieFactoryWithTxn.NewInternalExecutorWithTxn(sd, &cf.settings.SV, txn, descsCol) 
        
           if err := f(ctx, txn, descsCol, ie); err != nil { 
        
           	return err

in pkg/sql/catalog/descs.(*CollectionFactory).TxnWithExecutor.func3

cockroach/pkg/kv/db.go

Lines 974 to 976 in 77667a1

    
           err := txn.exec(ctx, func(ctx context.Context, txn *Txn) error { 
        
           	return retryable(ctx, txn) 
        
           })

in pkg/kv.runTxn.func1

cockroach/pkg/kv/txn.go

Lines 960 to 962 in 77667a1

    
           } 
        
           err = fn(ctx, txn)

in pkg/kv.(*Txn).exec

cockroach/pkg/kv/db.go

Lines 973 to 975 in 77667a1

    
           func runTxn(ctx context.Context, txn *Txn, retryable func(context.Context, *Txn) error) error { 
        
           	err := txn.exec(ctx, func(ctx context.Context, txn *Txn) error { 
        
           		return retryable(ctx, txn)

in pkg/kv.runTxn

cockroach/pkg/kv/db.go

Lines 936 to 938 in 77667a1

    
           	txn.SetDebugName("unnamed") 
        
           	return runTxn(ctx, txn, retryable) 
        
           }

in pkg/kv.(*DB).TxnWithAdmissionControl

cockroach/pkg/kv/db.go

Lines 915 to 917 in 77667a1

    
           func (db *DB) Txn(ctx context.Context, retryable func(context.Context, *Txn) error) error { 
        
           	return db.TxnWithAdmissionControl( 
        
           		ctx, roachpb.AdmissionHeader_OTHER, admissionpb.NormalPri, retryable)

in pkg/kv.(*DB).Txn

cockroach/pkg/sql/catalog/descs/txn.go

Lines 154 to 156 in 77667a1

    
           var deletedDescs catalog.DescriptorIDSet 
        
           if err := run(ctx, func(ctx context.Context, txn *kv.Txn) (err error) { 
        
           	withNewVersion, deletedDescs = nil, catalog.DescriptorIDSet{}

in pkg/sql/catalog/descs.(*CollectionFactory).TxnWithExecutor

cockroach/pkg/sql/catalog/descs/txn.go

Lines 48 to 50 in 77667a1

    
           ) error { 
        
           	return cf.TxnWithExecutor(ctx, db, nil /* sessionData */, func( 
        
           		ctx context.Context, txn *kv.Txn, descriptors *Collection, _ sqlutil.InternalExecutor,

in pkg/sql/catalog/descs.(*CollectionFactory).Txn

cockroach/pkg/sql/schema_changer.go

Lines 2463 to 2465 in 77667a1

    
           	} 
        
           	return sc.execCfg.CollectionFactory.Txn(ctx, sc.db, f) 
        
           }

in pkg/sql.(*SchemaChanger).txn

GOROOT/src/runtime/panic.go in runtime.gopanic at line 884
GOROOT/src/runtime/panic.go in runtime.panicmem at line 260
GOROOT/src/runtime/signal_unix.go in runtime.sigpanic at line 835
pkg/kv/txn.go in pkg/kv.(*Txn).CommitTimestamp at line 373
pkg/sql/sem/eval/context.go in pkg/sql/sem/eval.(*Context).GetClusterTimestamp at line 406
pkg/sql/sem/builtins/builtins.go in pkg/sql/sem/builtins.glob..func148 at line 2785
pkg/sql/sem/eval/expr.go in pkg/sql/sem/eval.(*evaluator).EvalFuncExpr at line 459
bazel-out/k8-opt/bin/pkg/sql/sem/tree/eval_expr_generated.go in pkg/sql/sem/tree.(*FuncExpr).Eval at line 264
pkg/sql/sem/eval/expr.go in pkg/sql/sem/eval.Expr at line 24
pkg/sql/backfill/backfill.go in pkg/sql/backfill.(*ColumnBackfiller).RunColumnBackfillChunk at line 370
pkg/sql/rowexec/columnbackfiller.go in pkg/sql/rowexec.(*columnBackfiller).runChunk.func1 at line 132
pkg/kv/db.go in pkg/kv.runTxn.func1 at line 975
pkg/kv/txn.go in pkg/kv.(*Txn).exec at line 961
pkg/kv/db.go in pkg/kv.runTxn at line 974
pkg/kv/db.go in pkg/kv.(*DB).TxnWithAdmissionControl at line 937
pkg/sql/rowexec/columnbackfiller.go in pkg/sql/rowexec.(*columnBackfiller).runChunk at line 114
pkg/sql/rowexec/backfiller.go in pkg/sql/rowexec.(*backfiller).mainLoop at line 154
pkg/sql/rowexec/backfiller.go in pkg/sql/rowexec.(*backfiller).doRun at line 107
pkg/sql/rowexec/backfiller.go in pkg/sql/rowexec.(*backfiller).Run at line 95
pkg/sql/flowinfra/flow.go in pkg/sql/flowinfra.(*FlowBase).Run at line 472
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).Run at line 695
pkg/sql/backfill.go in pkg/sql.(*SchemaChanger).distColumnBackfill.func3 at line 1348
pkg/sql/catalog/descs/txn.go in pkg/sql/catalog/descs.(*CollectionFactory).Txn.func1 at line 52
pkg/sql/catalog/descs/txn.go in pkg/sql/catalog/descs.(*CollectionFactory).TxnWithExecutor.func3 at line 163
pkg/kv/db.go in pkg/kv.runTxn.func1 at line 975
pkg/kv/txn.go in pkg/kv.(*Txn).exec at line 961
pkg/kv/db.go in pkg/kv.runTxn at line 974
pkg/kv/db.go in pkg/kv.(*DB).TxnWithAdmissionControl at line 937
pkg/kv/db.go in pkg/kv.(*DB).Txn at line 916
pkg/sql/catalog/descs/txn.go in pkg/sql/catalog/descs.(*CollectionFactory).TxnWithExecutor at line 155
pkg/sql/catalog/descs/txn.go in pkg/sql/catalog/descs.(*CollectionFactory).Txn at line 49
pkg/sql/schema_changer.go in pkg/sql.(*SchemaChanger).txn at line 2464

Tag	Value
Cockroach Release	`v22.2.0`
Cockroach SHA:	`77667a1`
Platform	linux amd64
Distribution	CCL
Environment	v22.2.0
Command	server
Go Version	``
# of CPUs
# of Goroutines

Jira issue: CRDB-22567

The text was updated successfully, but these errors were encountered:

ajwerner · 2023-01-17T14:48:49Z

Oof, you are not allowed to use cluster_logical_timestamp in a default expression. We need to find a better way to disallow it.

ajwerner · 2023-04-04T17:51:02Z

@Xiang-Gu did you do something about this?

rafiss · 2023-05-18T20:39:15Z

fixed by #99662

cockroach-teamcity added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. labels Dec 19, 2022

yuzefovich added this to Triage in SQL Foundations via automation Dec 27, 2022

blathers-crl bot added the T-sql-schema-deprecated Use T-sql-foundations instead label Dec 27, 2022

postamar assigned ajwerner Jan 17, 2023

postamar moved this from Triage to In Flight in SQL Foundations Jan 17, 2023

ajwerner removed their assignment Apr 4, 2023

exalate-issue-sync bot added T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) and removed T-sql-schema-deprecated Use T-sql-foundations instead labels May 10, 2023

rafiss closed this as completed May 18, 2023

SQL Foundations automation moved this from In Flight to Done May 18, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

backfill: v22.2.0: nil txn crash in cluster_logical_timestamp builtin in backfill #93870

backfill: v22.2.0: nil txn crash in cluster_logical_timestamp builtin in backfill #93870

cockroach-teamcity commented Dec 19, 2022 •

edited by cockroach-jira-scripts

Loading

ajwerner commented Jan 17, 2023

ajwerner commented Apr 4, 2023

rafiss commented May 18, 2023

backfill: v22.2.0: nil txn crash in cluster_logical_timestamp builtin in backfill #93870

backfill: v22.2.0: nil txn crash in cluster_logical_timestamp builtin in backfill #93870

Comments

cockroach-teamcity commented Dec 19, 2022 • edited by cockroach-jira-scripts Loading

ajwerner commented Jan 17, 2023

ajwerner commented Apr 4, 2023

rafiss commented May 18, 2023

cockroach-teamcity commented Dec 19, 2022 •

edited by cockroach-jira-scripts

Loading