pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload: TestWorkload failed

[cockroach-teamcity]

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload.TestWorkload failed with artifacts on master @ aa904152ef77a5af1b1e0df83fd7851f88c9ac51:

     "public",
     "table_w0_10",
     "col_w0_10_w1_134"
    ],
    "result": false
   }
  ],
  "previousStatements": [
   "ALTER TABLE public.table_w0_10 RENAME COLUMN col10_w0_33 TO col_w0_10_w1_134"
  ]
 }
}
    schema_change_external_test.go:165: 
        	Error Trace:	pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:165
        	Error:      	Received unexpected error:
        	            	***UNEXPECTED ERROR; Received an unexpected execution error.: ERROR: relation "table_w0_10" (109): index "table_w0_10_col10_w0_33_col10_w0_28_col10_w0_24_col10_w0_29_key" refers to non-existent shard column "crdb_internal_col10_w0_24_col10_w0_28_col10_w0_29_col_w0_10_w1_134_shard_16" (SQLSTATE XXUUU)
        	            	(1) forced error mark
        	            	  | "fatal error when running txn"
        	            	  | github.com/cockroachdb/errors/withstack/*withstack.withStack::
        	            	Wraps: (2)
        	            	Wraps: (3) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*opStmt).executeStmt
        	            	  | 	pkg/workload/schemachange/operation_generator.go:3385
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn
        	            	  | 	pkg/workload/schemachange/schemachange.go:613
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run
        	            	  | 	pkg/workload/schemachange/schemachange.go:704
        	            	  | pkg/ccl/testccl/workload/schemachange/schemachange_test_test.TestWorkload.TestWorkload.func5.func7
        	            	  | 	pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:154
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	external/org_golang_x_sync/errgroup/errgroup.go:93
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_arm64.s:1268
        	            	Wraps: (4) ***UNEXPECTED ERROR; Received an unexpected execution error.
        	            	Wraps: (5) ERROR: relation "table_w0_10" (109): index "table_w0_10_col10_w0_33_col10_w0_28_col10_w0_24_col10_w0_29_key" refers to non-existent shard column "crdb_internal_col10_w0_24_col10_w0_28_col10_w0_29_col_w0_10_w1_134_shard_16" (SQLSTATE XXUUU)
        	            	Error types: (1) *markers.withMark (2) *schemachange.ErrorState (3) *withstack.withStack (4) *errutil.withPrefix (5) *pgconn.PgError
        	Test:       	TestWorkload
Schema Workload Stats
Total Schema Statements Executed = 71
Total Schema Statements Succeeded = 71
Total Schema Statement Expected Failures = 0
Total Transactions Committed = 42
Total Transactions Rolled Back = 39
Total Transactions Executed = 81
    schema_change_external_test.go:118: backup, tracing data, and system table dumps in /artifacts/tmp/_tmp/67537770b84c3a4ea177539c1e46bd43/logTestWorkload2087710061
    schema_change_external_test.go:127: running DROP with use_declarative_schema_changer = on
    panic.go:615: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/67537770b84c3a4ea177539c1e46bd43/logTestWorkload2087710061
--- FAIL: TestWorkload (55.95s)

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-56241

[spilchen]

This one looks similar to #148822, but that was suppose to be fixed already.

[spilchen]

The above link is for a fix for DROP COLUMN. Let's investigate this as a RENAME COLUMN issue.

[blathers-crl]

The release-blocker label has been automatically removed because this issue was assigned priority P-2. Only P-0 and P-1 issues are considered release blockers.

[spilchen]

Here is a repro:

CREATE TABLE table_w0_10 (
      col10_w0_24 INT8      NOT NULL,
      col10_w0_28 DATE      NOT NULL,
      col10_w0_29 INT8      NOT NULL,
      col10_w0_33 INT8      NOT NULL,
      col10_w0_31 INT8,
      col10_w0_34 STRING[]  NOT NULL,
      PRIMARY KEY (col10_w0_33, col10_w0_28, col10_w0_24) USING HASH WITH BUCKET_COUNT = 16
  );

CREATE UNIQUE INDEX
  table_w0_10_col10_w0_33_col10_w0_28_col10_w0_24_col10_w0_29_key
  ON table_w0_10 (col10_w0_33, col10_w0_28, col10_w0_24,
  col10_w0_29)
  USING HASH WITH BUCKET_COUNT = 16;

ALTER TABLE table_w0_10
  RENAME COLUMN col10_w0_33 TO col_w0_10_w1_134;

The ALTER fails with:

ERROR: error executing StatementPhase stage 1 of 1 with 3 MutationType ops: relation "table_w0_10" (114): index "table_w0_10_col10_w0_33_col10_w0_28_col10_w0_24_col10_w0_29_key" refers to non-existent shard column "crdb_internal_col10_w0_24_col10_w0_28_col10_w0_29_col_w0_10_w1_134_shard_16"
DETAIL: • Schema change plan for ALTER TABLE repro.public.table_w0_10 RENAME COLUMN col10_w0_33 TO col_w0_10_w1_134

[spilchen]

The problem is when we have multiple shard columns. We aren't handling the rename of both columns. Our logic stops after renaming the first shard column. This is problem in the DSC and in the legacy schema changer.