sql: Panic after moving table from REGIONAL BY ROW to REGIONAL BY TABLE and back to REGIONAL BY ROW

[ajstorm]

When running the included script, ALTERs from REGIONAL BY ROW to REGIONAL BY TABLE, and back to REGIONAL BY ROW will succeed, but then eventually panic with the below stack. Hit this on ajstorm_zone_config_force_table branch, but it might also be reproducible on master.

zone_configs_repro.scr.zip

* ERROR: a panic has occurred!
* use of enum metadata before hydration as an enum
* (1) attached stack trace
*   -- stack trace:
*   | runtime.gopanic
*   | 	/usr/local/opt/go/libexec/src/runtime/panic.go:969
*   | [...repeated from below...]
* Wraps: (2) assertion failure
* Wraps: (3) attached stack trace
*   -- stack trace:
*   | github.com/cockroachdb/cockroach/pkg/sql/types.(*T).ensureHydratedEnum
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/types/types.go:2415
*   | github.com/cockroachdb/cockroach/pkg/sql/types.(*T).EnumGetIdxOfPhysical
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/types/types.go:2381
*   | github.com/cockroachdb/cockroach/pkg/sql/sem/tree.GetEnumComponentsFromPhysicalRep
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/sem/tree/datum.go:3977
*   | github.com/cockroachdb/cockroach/pkg/sql/rowenc.DecodeUntaggedDatum
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowenc/column_type_encoding.go:674
*   | github.com/cockroachdb/cockroach/pkg/sql/rowenc.DecodeTableValue
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowenc/column_type_encoding.go:524
*   | github.com/cockroachdb/cockroach/pkg/sql/rowenc.DecodePartitionTuple
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowenc/partition.go:146
*   | github.com/cockroachdb/cockroach/pkg/sql.indexCoveringsForPartitioning
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/partition_utils.go:202
*   | github.com/cockroachdb/cockroach/pkg/sql.GenerateSubzoneSpans.func1
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/partition_utils.go:118
*   | github.com/cockroachdb/cockroach/pkg/sql/catalog.ForEachIndex
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/catalog/descriptor.go:613
*   | github.com/cockroachdb/cockroach/pkg/sql.GenerateSubzoneSpans
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/partition_utils.go:103
*   | github.com/cockroachdb/cockroach/pkg/sql.writeZoneConfig
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/set_zone_config.go:903
*   | github.com/cockroachdb/cockroach/pkg/sql.RemoveIndexZoneConfigs
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/set_zone_config.go:989
*   | github.com/cockroachdb/cockroach/pkg/sql/gcjob.gcIndexes.func2
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/gcjob/index_garbage_collection.go:69
*   | github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn.func1
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:772
*   | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn.go:828
*   | github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:771
*   | github.com/cockroachdb/cockroach/pkg/sql/gcjob.gcIndexes
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/gcjob/index_garbage_collection.go:68
*   | github.com/cockroachdb/cockroach/pkg/sql/gcjob.performGC
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/gcjob/gc_job.go:66
*   | github.com/cockroachdb/cockroach/pkg/sql/gcjob.schemaChangeGCResumer.Resume
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/gcjob/gc_job.go:158
*   | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func1
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1178
*   | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1179
*   | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:255
*   | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).resumeJob.func1
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:214
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351
*   | runtime.goexit
*   | 	/usr/local/opt/go/libexec/src/runtime/asm_amd64.s:1374
* Wraps: (4) use of enum metadata before hydration as an enum
* Error types: (1) *withstack.withStack (2) *assert.withAssertionFailure (3) *withstack.withStack (4) *errutil.leafError
*
panic: use of enum metadata before hydration as an enum [recovered]
	panic: use of enum metadata before hydration as an enum

goroutine 48849 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Recover(0xc000557d80, 0x943d0a0, 0xc0015362c0)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:233 +0x126
panic(0x8196600, 0xc014f7ac80)
	/usr/local/opt/go/libexec/src/runtime/panic.go:969 +0x1b9
github.com/cockroachdb/cockroach/pkg/sql/types.(*T).ensureHydratedEnum(...)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/types/types.go:2415
github.com/cockroachdb/cockroach/pkg/sql/types.(*T).EnumGetIdxOfPhysical(0xc0109f7680, 0xc006d6434e, 0x1, 0x1, 0x1, 0x0, 0x0)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/types/types.go:2381 +0x426
github.com/cockroachdb/cockroach/pkg/sql/sem/tree.GetEnumComponentsFromPhysicalRep(0xc0109f7680, 0xc006d6434e, 0x1, 0x1, 0x0, 0x0, 0xc006d6434e, 0x1, 0x1, 0x0, ...)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/sem/tree/datum.go:3977 +0x4d
github.com/cockroachdb/cockroach/pkg/sql/rowenc.DecodeUntaggedDatum(0xc00768ce10, 0xc0109f7680, 0xc006d6434d, 0x2, 0x2, 0x0, 0x6, 0x0, 0x0, 0x2, ...)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowenc/column_type_encoding.go:674 +0x3385
github.com/cockroachdb/cockroach/pkg/sql/rowenc.DecodeTableValue(0xc00768ce10, 0xc0109f7680, 0xc006d6434c, 0x3, 0x3, 0x0, 0x6, 0x0, 0x0, 0x8, ...)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowenc/column_type_encoding.go:524 +0x165
github.com/cockroachdb/cockroach/pkg/sql/rowenc.DecodePartitionTuple(0xc00768ce10, 0xc0017be500, 0xc0017be500, 0x0, 0x95ac880, 0xc0012c7200, 0xc0012c7288, 0xc0012c73c8, 0xc006d6434c, 0x3, ...)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowenc/partition.go:146 +0x492
github.com/cockroachdb/cockroach/pkg/sql.indexCoveringsForPartitioning(0xc00768ce10, 0xc0017be500, 0xc0017be500, 0x0, 0x95ac880, 0xc0012c7200, 0xc0012c7288, 0xc0012c73c8, 0xc00768c8b0, 0x0, ...)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/partition_utils.go:202 +0xb3f
github.com/cockroachdb/cockroach/pkg/sql.GenerateSubzoneSpans.func1(0x959dc60, 0xc0120a5c40, 0x10000, 0x959dc60)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/partition_utils.go:118 +0x1ae
github.com/cockroachdb/cockroach/pkg/sql/catalog.ForEachIndex(0x95ac880, 0xc0012c7200, 0xc006010000, 0xc00768c9c8, 0xc00768cab0, 0xc00de56800)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/catalog/descriptor.go:613 +0x11f
github.com/cockroachdb/cockroach/pkg/sql.GenerateSubzoneSpans(0xc0007ed500, 0xff49ba3b6e8caeb5, 0x6f341dfbc6c164a5, 0xc0017be500, 0xc0017be500, 0x0, 0x95ac880, 0xc0012c7200, 0xc005dc7c00, 0x6, ...)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/partition_utils.go:103 +0x375
github.com/cockroachdb/cockroach/pkg/sql.writeZoneConfig(0x943d160, 0xc011f93290, 0xc0036ff830, 0xc000000040, 0x95ac880, 0xc0012c7200, 0xc0109f78c0, 0xc001594000, 0x0, 0x0, ...)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/set_zone_config.go:903 +0x498
github.com/cockroachdb/cockroach/pkg/sql.RemoveIndexZoneConfigs(0x943d160, 0xc011f93290, 0xc0036ff830, 0xc001594000, 0x40, 0xc00768d670, 0x1, 0x1, 0x0, 0xc0007ed500)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/set_zone_config.go:989 +0x2d5
github.com/cockroachdb/cockroach/pkg/sql/gcjob.gcIndexes.func2(0x943d160, 0xc011f93290, 0xc0036ff830, 0x0, 0x0)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/gcjob/index_garbage_collection.go:69 +0x111
github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn.func1(0x943d160, 0xc011f93290, 0xc0036ff830, 0x3ff0000000000000, 0x9591000)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:772 +0x43
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec(0xc0036ff830, 0x943d160, 0xc011f93290, 0xc00768dbe0, 0xc0036ff830, 0x7ebfc00)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn.go:828 +0xdc
github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn(0xc0010ceb60, 0x943d160, 0xc011f93290, 0xc00768dee0, 0xc00781c900, 0x0)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:771 +0x105
github.com/cockroachdb/cockroach/pkg/sql/gcjob.gcIndexes(0x943d160, 0xc011f93290, 0xc001594000, 0x40, 0xc0174b5d00, 0xc0100ff550, 0x76170c0)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/gcjob/index_garbage_collection.go:68 +0x31a
github.com/cockroachdb/cockroach/pkg/sql/gcjob.performGC(0x943d160, 0xc011f93290, 0xc001594000, 0xc01889ecc0, 0xc0174b5d00, 0xc00a3a6740, 0x28)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/gcjob/gc_job.go:66 +0x225
github.com/cockroachdb/cockroach/pkg/sql/gcjob.schemaChangeGCResumer.Resume(0x8e118e573060001, 0x943d160, 0xc011f93290, 0x82a5600, 0xc004c84ee0, 0x0, 0x0)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/gcjob/gc_job.go:158 +0x578
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func1(0xc00159e0c0, 0x93e7720, 0xc00ac46af8, 0x943d160, 0xc011f93290, 0x82a5600, 0xc004c84ee0, 0xc00768fa50)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1178 +0xae
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine(0xc000789040, 0x943d160, 0xc011f93230, 0x82a5600, 0xc004c84ee0, 0x93e7720, 0xc00ac46af8, 0xc0126f72c0, 0xc002526901, 0x7, ...)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1179 +0x885
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob(0xc000789040, 0x943d0a0, 0xc018c879c0, 0x93e7720, 0xc00ac46af8, 0xc0126f72c0, 0xc002526901, 0x7, 0xc00d7be540, 0x16, ...)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:255 +0x378
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).resumeJob.func1(0x943d0a0, 0xc0015362c0)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:214 +0x165
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000557d80, 0x943d0a0, 0xc0015362c0, 0x0, 0xc012fafb30)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc```

[blathers-crl]

Hi @ajstorm, please add branch-* labels to identify which branch(es) this release-blocker affects.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.}

[ajstorm]

zone_configs_repro.scr.zip

Simpler repro.

[ajstorm]

Fix coming soon, but here's a writeup of what's going on. The transition from REGIONAL BY ROW to REGIONAL BY TABLE drops the partitioning on the table but does not drop the zone configurations. As a result, when the index GC runs, it will try and cleanup the zone configurations. This hits an issue though, as the types are not properly hydrated in the index GC code path. As a result, to work around this problem we should add code to remove the zone configs when dropping the table partitioning. This will not solve the root cause of this issue, but it will instead prevent the failure from occurring when we're moving from RBR->RBT.

A new issue will be opened soon to track the more general problem.