workload/schemachange: running workload with same seed generates different schema identifiers

[renatolabs]

Running the schemachange workload with the same seed (with COCKROACH_RANDOM_SEED=x) produces the same operations, but with different identifiers. For example, at the time of writing, these are the first two operations generated by running the following command two times:

COCKROACH_RANDOM_SEED=9082679504586840611 ./bin/workload run schemachange --concurrency 2 --max-ops 10 --verbose 1

Run 1:

{
 "workerId": 1,
 "clientTimestamp": "20:42:05.993818",
 "ops": [
  "BEGIN",
  {
   "sql": "CREATE INDEX \"IrrelevantName\" ON public.table2 (\"IrrelevantColumn\" ASC)",
   "expectedExecErr": "42P01"
  }
 ],
 "expectedExecErrors": "42P01",
 "expectedCommitErrors": "",
 "message": "ROLLBACK; Successfully got expected execution error.: ERROR: relation \"public.table2\" does not exist (SQLSTATE 42P01)"
}
{
 "workerId": 0,
 "clientTimestamp": "20:42:05.993707",
 "ops": [
  "BEGIN",
  {
   "sql": "DROP TABLE public.table3",
   "expectedExecErr": "42P01"
  }
 ],
 "expectedExecErrors": "42P01",
 "expectedCommitErrors": "",
 "message": "ROLLBACK; Successfully got expected execution error.: ERROR: relation \"public.table3\" does not exist (SQLSTATE 42P01)"
}

Run 2:

{
 "workerId": 1,
 "clientTimestamp": "20:42:13.542074",
 "ops": [
  "BEGIN",
  {
   "sql": "CREATE INDEX \"IrrelevantName\" ON public.table7 (\"IrrelevantColumn\" ASC)",
   "expectedExecErr": "42P01"
  }
 ],
 "expectedExecErrors": "42P01",
 "expectedCommitErrors": "",
 "message": "ROLLBACK; Successfully got expected execution error.: ERROR: relation \"public.table7\" does not exist (SQLSTATE 42P01)"
}
{
 "workerId": 0,
 "clientTimestamp": "20:42:13.542037",
 "ops": [
  "BEGIN",
  {
   "sql": "DROP TABLE public.table8",
   "expectedExecErr": "42P01"
  }
 ],
 "expectedExecErrors": "42P01",
 "expectedCommitErrors": "",
 "message": "ROLLBACK; Successfully got expected execution error.: ERROR: relation \"public.table8\" does not exist (SQLSTATE 42P01)"
}

There seems to be some other form of non determinism outside the seed passed that makes the identifiers different. Having schemachange generate the same identifiers has at least two big advantages:

• increases the chances of us being able to reproduce failures, especially in cases where the failure happens due to interactions of several calls to the schemachange workload (for example, the failure observed in roachtest: set random seed in schemachange step of version-upgrade #117409).
• ease of debugging: it's a better debugging experience if we are able to run the test with the same seed and scan the output generated by the workload if the identifiers are identical.

Jira issue: CRDB-35168

[rafiss]

#117332 may have addressed this. @renatolabs do you have an easy way to check?

[renatolabs]

@renatolabs do you have an easy way to check?

Easiest way, IMO, would be to run the workload with the same seed and comparing, as I did in the issue.

I ran the commands above again on master (2c3b07e) and I still see different identifiers.

[rafiss]

Ah I see, thanks.

It looks like the names are generated here:

cockroach/pkg/workload/schemachange/operation_generator.go

Lines 3580 to 3585 in 6bf526e

	if og.randIntn(100) >= pctExisting {
	treeTableName := tree.MakeTableNameFromPrefix(tree.ObjectNamePrefix{
	SchemaName: tree.Name(desiredSchema),
	ExplicitSchema: true,
	}, tree.Name(fmt.Sprintf("table%d", og.newUniqueSeqNum())))
	return &treeTableName, nil

The problem is that part of the name is og.newUniqueSeqNum() , which is just a counter shared between all the workers, and it doesn't use the RNG. So that means the names are dependent on goroutine scheduling.

We can't just make seqNum unique per worker, since then that could mean different workers would both try to create a table with the same name. But perhaps we can make the new table name be of the format table_{workerID}_{seqNum}

[renatolabs]

we can make the new table name be of the format table_{workerID}_{seqNum}

Yeah, I like that option!