schema: pause schema change job when running out of disk space

[rimadeodhar]

Currently, we pause a schema change job when its running out of disk space during an operation involving an index backfill.

Ask: Can we expand this functionality to column backfill as well?

This will help prevent a schema change from consuming all disk space and causing a cluster outage particularly on underprovisioned systems. See this support ticket for an example of such a case.

Jira issue: CRDB-39494

[rimadeodhar]

@fqazi - what do you think of the above approach? We should be able to expand this for other schema changes involving backfill as well right?

[fqazi]

I think its a reasonable approach, and I think was probably not there because it seems far less likely. We probably didn't expect the column backfill to balloon the size too much with an in place backfill.

[rimadeodhar]

I think there is a very real possibility of column backfill leading to this scenario. In the legacy case, we will rewrite all the rows containing the new column which will easily 2x the table size until GC runs. With the declarative schema changer, we simply recreate the index so once again the space usage is high. I think this is worth prioritizing as it will at least prevent cluster outages due to the disk filling up.

[rafiss]

Let's add this check for the column backfiller here:

cockroach/pkg/sql/backfill.go

Lines 2393 to 2397 in d7f4220

	uint64(columnBackfillUpdateChunkSizeThresholdBytes.Get(&sc.settings.SV)),
	backfill.ColumnMutationFilter,
	); err != nil {
	return err
	}

And also in the declarative schema changer here:

cockroach/pkg/sql/schemachanger/scexec/exec_backfill.go

Line 47 in 379a702

return runBackfiller(ctx, deps, tracker, backfillProgresses, mergeProgresses, deps.TransactionalJobRegistry().CurrentJob(), tables)