opt: v23.1.14: partial index predicate does not exist in table metadata

[cockroach-sentry]

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/4954614362/?referrer=webhooks_plugin

Panic Message:

table_meta.go:443: partial index predicate does not exist in table metadata
(1) assertion failure
Wraps: (2) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/cockroach/pkg/sql/opt.(*TableMeta).PartialIndexPredicate
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/table_meta.go:443
  | github.com/cockroachdb/cockroach/pkg/sql/opt/xform.(*scanIndexIter).ForEachStartingAfter
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/xform/scan_index_iter.go:246
  | github.com/cockroachdb/cockroach/pkg/sql/opt/xform.(*scanIndexIter).ForEach
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/xform/scan_index_iter.go:208
  | github.com/cockroachdb/cockroach/pkg/sql/opt/xform.(*CustomFuncs).GenerateIndexScans
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/xform/scan_funcs.go:51
  | github.com/cockroachdb/cockroach/pkg/sql/opt/xform.(*explorer).exploreScan
  | 	github.com/cockroachdb/cockroach/bazel-out/k8-opt/bin/pkg/sql/opt/xform/explorer.og.go:159
  | github.com/cockroachdb/cockroach/pkg/sql/opt/xform.(*explorer).exploreGroupMember
  | 	github.com/cockroachdb/cockroach/bazel-out/k8-opt/bin/pkg/sql/opt/xform/explorer.og.go:22
  | github.com/cockroachdb/cockroach/pkg/sql/opt/xform.(*explorer).exploreGroup
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/xform/explorer.go:185
  | github.com/cockroachdb/cockroach/pkg/sql/opt/xform.(*Optimizer).optimizeGroup
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/xform/optimizer.go:536
  | github.com/cockroachdb/cockroach/pkg/sql/opt/xform.(*Optimizer).optimizeExpr
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/xform/optimizer.go:299
  | github.com/cockroachdb/cockroach/pkg/sql/opt/xform.(*Optimizer).optimizeGroupMember
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/xform/optimizer.go:578
  | github.com/cockroachdb/cockroach/pkg/sql/opt/xform.(*Optimizer).optimizeGroup
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/xform/optimizer.go:523
  | github.com/cockroachdb/cockroach/pkg/sql/opt/xform.(*Optimizer).Optimize
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/xform/optimizer.go:269
  | github.com/cockroachdb/cockroach/pkg/sql.(*optPlanningCtx).buildExecMemo
  | 	github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:598
  | github.com/cockroachdb/cockroach/pkg/sql.(*planner).makeOptimizerPlan
  | 	github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:245
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).makeExecPlan
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:2060
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1567
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1037
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:142
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithProfiling
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:3119
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:141
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2183
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2188
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2105
  | github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:902
  | github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:310
  | runtime.goexit
  | 	src/runtime/asm_arm64.s:1172
Wraps: (3) partial index predicate does not exist in table metadata
Error types: (1) *assert.withAssertionFailure (2) *withstack.withStack (3) *errutil.leafError
-- report composition:
*errutil.leafError: partial index predicate does not exist in table metadata
table_meta.go:443: *withstack.withStack (top exception)
*assert.withAssertionFailure

Stacktrace (expand for inline code snippets):

src/runtime/asm_arm64.s#L1171-L1173

cockroach/pkg/sql/pgwire/conn.go

Lines 309 to 311 in b95f222

	reservedOwned = false // We're about to pass ownership away.
	retErr = sqlServer.ServeConn(
	ctx,

cockroach/pkg/sql/conn_executor.go

Lines 901 to 903 in b95f222

	}(ctx, h)
	return h.ex.run(ctx, s.pool, reserved, cancel)
	}

cockroach/pkg/sql/conn_executor.go

Lines 2104 to 2106 in b95f222

	var err error
	if err = ex.execCmd(); err != nil {
	// Both of these errors are normal ways for the connExecutor to exit.

cockroach/pkg/sql/conn_executor.go

Lines 2187 to 2189 in b95f222

	return err
	}()
	// Note: we write to ex.statsCollector.PhaseTimes, instead of ex.phaseTimes,

cockroach/pkg/sql/conn_executor.go

Lines 2182 to 2184 in b95f222

	tcmd.LastInBatch || !implicitTxnForBatch)
	ev, payload, err = ex.execStmt(
	ctx, tcmd.Statement, nil /* portal */, nil /* pinfo */, stmtRes, canAutoCommit,

cockroach/pkg/sql/conn_executor_exec.go

Lines 140 to 142 in b95f222

	}
	err = ex.execWithProfiling(ctx, ast, preparedStmt, func(ctx context.Context) error {
	ev, payload, err = ex.execStmtInOpenState(ctx, parserStmt, portal, pinfo, res, canAutoCommit)

cockroach/pkg/sql/conn_executor_exec.go

Lines 3118 to 3120 in b95f222

	} else {
	err = op(ctx)
	}

cockroach/pkg/sql/conn_executor_exec.go

Lines 141 to 143 in b95f222

	err = ex.execWithProfiling(ctx, ast, preparedStmt, func(ctx context.Context) error {
	ev, payload, err = ex.execStmtInOpenState(ctx, parserStmt, portal, pinfo, res, canAutoCommit)
	return err

cockroach/pkg/sql/conn_executor_exec.go

Lines 1036 to 1038 in b95f222

	if err = ex.dispatchToExecutionEngine(stmtCtx, p, res); err != nil {
	stmtThresholdSpan.Finish()

cockroach/pkg/sql/conn_executor_exec.go

Lines 1566 to 1568 in b95f222

	// between here and there needs to happen even if there's an error.
	err = ex.makeExecPlan(ctx, planner)
	defer planner.curPlan.close(ctx)

cockroach/pkg/sql/conn_executor_exec.go

Lines 2059 to 2061 in b95f222

	func (ex *connExecutor) makeExecPlan(ctx context.Context, planner *planner) error {
	if err := planner.makeOptimizerPlan(ctx); err != nil {
	log.VEventf(ctx, 1, "optimizer plan failed: %v", err)

cockroach/pkg/sql/plan_opt.go

Lines 244 to 246 in b95f222

	execMemo, err := opc.buildExecMemo(ctx)
	if err != nil {

cockroach/pkg/sql/plan_opt.go

Lines 597 to 599 in b95f222

	if _, isCanned := opc.p.stmt.AST.(*tree.CannedOptPlan); !isCanned {
	if _, err := opc.optimizer.Optimize(); err != nil {
	return nil, err

cockroach/pkg/sql/opt/xform/optimizer.go

Lines 268 to 270 in b95f222

	rootProps := o.mem.RootProps()
	o.optimizeGroup(root, rootProps)

cockroach/pkg/sql/opt/xform/optimizer.go

Lines 522 to 524 in b95f222

	// Optimize the group member with respect to the required properties.
	memberOptimized := o.optimizeGroupMember(state, member, required)

cockroach/pkg/sql/opt/xform/optimizer.go

Lines 577 to 579 in b95f222

	// Optimize the child with respect to those properties.
	childCost, childOptimized := o.optimizeExpr(member.Child(i), childRequired)

cockroach/pkg/sql/opt/xform/optimizer.go

Lines 298 to 300 in b95f222

	case memo.RelExpr:
	state := o.optimizeGroup(t, required)
	return state.cost, state.fullyOptimized

cockroach/pkg/sql/opt/xform/optimizer.go

Lines 535 to 537 in b95f222

	// other expressions in this group.
	if o.shouldExplore(required) && !o.explorer.exploreGroup(grp, required).fullyExplored {
	fullyOptimized = false

cockroach/pkg/sql/opt/xform/explorer.go

Lines 184 to 186 in b95f222

	if memberExplored := e.exploreGroupMember(state, member, i, required); memberExplored {
	// No more rules can ever match this expression, so skip it in

https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/bazel-out/k8-opt/bin/pkg/sql/opt/xform/explorer.og.go#L21-L23
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/bazel-out/k8-opt/bin/pkg/sql/opt/xform/explorer.og.go#L158-L160

cockroach/pkg/sql/opt/xform/scan_funcs.go

Lines 50 to 52 in b95f222

	iter.Init(c.e.evalCtx, c.e.f, c.e.mem, &c.im, scanPrivate, nil /* filters */, rejectPrimaryIndex|rejectInvertedIndexes)
	iter.ForEach(func(index cat.Index, filters memo.FiltersExpr, indexCols opt.ColSet, isCovering bool, constProj memo.ProjectionsExpr) {
	// The iterator only produces pseudo-partial indexes (the predicate is

cockroach/pkg/sql/opt/xform/scan_index_iter.go

Lines 207 to 209 in b95f222

	func (it *scanIndexIter) ForEach(f enumerateIndexFunc) {
	it.ForEachStartingAfter(cat.PrimaryIndex-1, f)
	}

cockroach/pkg/sql/opt/xform/scan_index_iter.go

Lines 245 to 247 in b95f222

	pred, isPartialIndex := it.tabMeta.PartialIndexPredicate(ord)

cockroach/pkg/sql/opt/table_meta.go

Lines 442 to 444 in b95f222

	if !ok {
	panic(errors.AssertionFailedf("partial index predicate does not exist in table metadata"))
	}

src/runtime/asm_arm64.s in runtime.goexit at line 1172
pkg/sql/pgwire/conn.go in pkg/sql/pgwire.(*conn).processCommandsAsync.func1 at line 310
pkg/sql/conn_executor.go in pkg/sql.(*Server).ServeConn at line 902
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).run at line 2105
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd at line 2188
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd.func1 at line 2183
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt at line 141
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execWithProfiling at line 3119
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt.func1 at line 142
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmtInOpenState at line 1037
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).dispatchToExecutionEngine at line 1567
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).makeExecPlan at line 2060
pkg/sql/plan_opt.go in pkg/sql.(*planner).makeOptimizerPlan at line 245
pkg/sql/plan_opt.go in pkg/sql.(*optPlanningCtx).buildExecMemo at line 598
pkg/sql/opt/xform/optimizer.go in pkg/sql/opt/xform.(*Optimizer).Optimize at line 269
pkg/sql/opt/xform/optimizer.go in pkg/sql/opt/xform.(*Optimizer).optimizeGroup at line 523
pkg/sql/opt/xform/optimizer.go in pkg/sql/opt/xform.(*Optimizer).optimizeGroupMember at line 578
pkg/sql/opt/xform/optimizer.go in pkg/sql/opt/xform.(*Optimizer).optimizeExpr at line 299
pkg/sql/opt/xform/optimizer.go in pkg/sql/opt/xform.(*Optimizer).optimizeGroup at line 536
pkg/sql/opt/xform/explorer.go in pkg/sql/opt/xform.(*explorer).exploreGroup at line 185
bazel-out/k8-opt/bin/pkg/sql/opt/xform/explorer.og.go in pkg/sql/opt/xform.(*explorer).exploreGroupMember at line 22
bazel-out/k8-opt/bin/pkg/sql/opt/xform/explorer.og.go in pkg/sql/opt/xform.(*explorer).exploreScan at line 159
pkg/sql/opt/xform/scan_funcs.go in pkg/sql/opt/xform.(*CustomFuncs).GenerateIndexScans at line 51
pkg/sql/opt/xform/scan_index_iter.go in pkg/sql/opt/xform.(*scanIndexIter).ForEach at line 208
pkg/sql/opt/xform/scan_index_iter.go in pkg/sql/opt/xform.(*scanIndexIter).ForEachStartingAfter at line 246
pkg/sql/opt/table_meta.go in pkg/sql/opt.(*TableMeta).PartialIndexPredicate at line 443

Tags

Tag	Value
Command	start-single-node
Environment	v23.1.14
Go Version	go1.19.13
Platform	linux arm64
Distribution	CCL
Cockroach Release	v23.1.14
Cockroach SHA	b95f222
# of CPUs	10
# of Goroutines	407

Jira issue: CRDB-35957

[XiaochenCui]

Hi, @yuzefovich
Are you investigating this issue? I would like to debug it if no one is working on it.

[yuzefovich]

@XiaochenCui thanks for your interest! I don't think anyone is actively looking into this, so please feel free to take a stab.

Note, though, the automatically-filed sentry issues don't have a reproduction, so debugging this type of issue usually involves some trial and error in hopes of getting a repro while reading the relevant code to get ideas on what could have gone wrong. (And we often don't find anything, so the issue is closed as unactionable. Still, we try to add some more debugging information if possible to the code, like a better error message.)

[XiaochenCui]

The lack of reproduction doesn't matter, it's normal in db domain.

The only issue is I'm new to crdb and may give my feedback for this issue in a long time (like 2 months). Do you think it's ok? And I don't have the permission for sentry.io

[yuzefovich]

Yeah, totally, there is definitely no rush in looking into this (as you can see, this issue has stayed in the "triage" column for over 2 months at this point).

Re: access to sentry.io: I don't think that we'll be able to share access to sentry.io with you, but also the sentry event doesn't really contain any more debugging information other than what's included on this github issue (i.e. we only have a stack trace and the corresponding CRDB version). (I looked at the event for #124240 since the sentry event for #118913 seems to have been deleted.)

[XiaochenCui]

@yuzefovich Thanks for your information