Skip to content

bulkio: enable elastic CPU control for index backfill by default#163866

Merged
trunk-io[bot] merged 1 commit intocockroachdb:masterfrom
dt:enable-index-backfill-elastic-control
Mar 4, 2026
Merged

bulkio: enable elastic CPU control for index backfill by default#163866
trunk-io[bot] merged 1 commit intocockroachdb:masterfrom
dt:enable-index-backfill-elastic-control

Conversation

@dt
Copy link
Contributor

@dt dt commented Feb 18, 2026

This change enables the bulkio.index_backfill.elastic_control.enabled cluster setting by default, integrating index backfill operations with elastic CPU control. This allows index backfill workloads to automatically throttle based on available CPU resources, improving cluster stability during schema changes.

Release note (ops change): The bulkio.index_backfill.elastic_control.enabled cluster setting is now enabled by default, allowing index backfill operations to integrate with elastic CPU control and automatically throttle based on available resources.

Epic: CRDB-48845.

@dt dt requested a review from rafiss February 18, 2026 18:19
@dt dt requested a review from a team as a code owner February 18, 2026 18:19
@dt dt requested review from DrewKimball and removed request for a team February 18, 2026 18:19
@trunk-io
Copy link
Contributor

trunk-io bot commented Feb 18, 2026

😎 Merged directly without going through the merge queue, as the queue was empty and the PR was up to date with the target branch - details.

@blathers-crl
Copy link

blathers-crl bot commented Feb 18, 2026

It looks like your PR touches production code but doesn't add or edit any test code. Did you consider adding tests to your PR?

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@cockroach-teamcity
Copy link
Member

This change is Reviewable

@dt dt force-pushed the enable-index-backfill-elastic-control branch from e770548 to 4d7c977 Compare February 18, 2026 18:21
@rafiss
Copy link
Collaborator

rafiss commented Feb 25, 2026

thanks for making this! i saw a timeout, not sure if it's related?

panic: test timed out after 5m0s
running tests:
TestVectorIndexMergingDuringBackfillWithPrefix (3m1s)

has this already been through some testing, or should we keep an eye out on the relevant roachperf tests after this merges?

@dt
Copy link
Contributor Author

dt commented Feb 25, 2026

TestVectorIndexMergingDuringBackfillWithPrefix ... related?

Yeah, very likely related work that steps aside under CPU saturation gets starved under prolonged/sustained total CPU saturation. That's by design for production systems but our CIs run under extreme CPU saturation conditions (since you just want all the tests to complete ASAP) so load-based AC doesn't play so well in intentionally overloaded test runners. Thus #163884 which was opened in parallel with this; now that that one has landed, I can give this one a rebase on that and can start putting DisableElasticCPUAdmission in the observed flakes.

roachperf

I did most of the benchmarking on IMPORT rather than backfiller and while they're broadly similar -- i.e. both loops generate KV pairs that go onto a channel that drains into a sst writer/addsstable -- I didn't do as much benchmarking of this path, so yeah, I think we'd want to watch roachperf. I would expect backfills that race with a workload to maybe get slightly slower -- by design -- but those that are uncontended shouldn't be slowed much, if at all.

@dt dt force-pushed the enable-index-backfill-elastic-control branch 2 times, most recently from b523db9 to 7a5e69d Compare February 26, 2026 00:44
Copy link
Collaborator

@rafiss rafiss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for pushing this through!

if you're able to do it before merging, could you update the elastic=false variant of the schemachange/bulkingest test in pkg/cmd/roachtest/tests/schemachange.go so that it sets bulkio.index_backfill.elastic_control.enabled to false?

@dt dt force-pushed the enable-index-backfill-elastic-control branch from 7a5e69d to 53beaf3 Compare March 4, 2026 16:21
This change enables the bulkio.index_backfill.elastic_control.enabled
cluster setting by default, integrating index backfill operations with
elastic CPU control. This allows index backfill workloads to automatically
throttle based on available CPU resources, improving cluster stability
during schema changes.

Release note (ops change): The bulkio.index_backfill.elastic_control.enabled
cluster setting is now enabled by default, allowing index backfill operations
to integrate with elastic CPU control and automatically throttle based on
available resources.

Epic: CRDB-48845.
@dt dt force-pushed the enable-index-backfill-elastic-control branch from 53beaf3 to 8177267 Compare March 4, 2026 17:15
@dt
Copy link
Contributor Author

dt commented Mar 4, 2026

update the elastic=false variant of the schemachange/bulkingest

done!

TFTR!

/trunk merge

@trunk-io trunk-io bot merged commit e7ff68e into cockroachdb:master Mar 4, 2026
24 of 25 checks passed
@dt dt deleted the enable-index-backfill-elastic-control branch March 4, 2026 18:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants