Skip to content

sql/schemachanger: slow rollout of distributed merge in Cloud via RAFA #167409

@rafiss

Description

@rafiss

Summary:
Use RAFA to do a slow rollout of the bulkio.index_backfill.distributed_merge.mode cluster setting on Cloud clusters during the 26.3 dev cycle. This will give us real-world signal on whether the feature works correctly before switching it on by default.

From the 26.2 pre-mortem.

Cluster setting: bulkio.index_backfill.distributed_merge.mode

  • Current default: disabled
  • Target value for rollout: enabled
  • Defined in pkg/sql/backfill/distributed_merge_mode.go

Precedent:
We followed a similar process for sql.defaults.autocommit_before_ddl.enabled in #133180 and https://github.com/cockroachlabs/rafa-production/pull/512.

Rollout plan:

  1. Wait for a 26.2 patch release to be deployed in CockroachDB Cloud.
  2. Ensure the SLI dashboard charts are in place (see related issue for dashboard work).
  3. Create a RAFA PR to set bulkio.index_backfill.distributed_merge.mode = 'enabled' on an initial set of high-risk clusters.
  4. Monitor the SLI dashboard and Datadog logs for the setting change.
  5. If no issues after ~1 week, expand to a broader set of clusters.
  6. Once confident, change the hard-coded default from disabled to enabled in pkg/sql/backfill/distributed_merge_mode.go.

Next Steps:

  • Create the RAFA PR targeting an initial cluster set
  • Monitor for regressions after each rollout stage
  • Change the hard-coded default once rollout is validated

Jira issue: CRDB-62441

Epic CRDB-62564

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-schema-changer-implRelated to the implementation of the new schema changerC-enhancementSolution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)O-agentFiled by an AI agent; usually the result of a human/agent investigation sessionO-premortemIssues identified during premortem exercise.T-sql-foundationsSQL Foundations Team (formerly SQL Schema + SQL Sessions)

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions