bulkio: enable elastic CPU control for index backfill by default#163866
Conversation
|
😎 Merged directly without going through the merge queue, as the queue was empty and the PR was up to date with the target branch - details. |
|
It looks like your PR touches production code but doesn't add or edit any test code. Did you consider adding tests to your PR? 🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
e770548 to
4d7c977
Compare
|
thanks for making this! i saw a timeout, not sure if it's related? has this already been through some testing, or should we keep an eye out on the relevant roachperf tests after this merges? |
Yeah, very likely related work that steps aside under CPU saturation gets starved under prolonged/sustained total CPU saturation. That's by design for production systems but our CIs run under extreme CPU saturation conditions (since you just want all the tests to complete ASAP) so load-based AC doesn't play so well in intentionally overloaded test runners. Thus #163884 which was opened in parallel with this; now that that one has landed, I can give this one a rebase on that and can start putting
I did most of the benchmarking on IMPORT rather than backfiller and while they're broadly similar -- i.e. both loops generate KV pairs that go onto a channel that drains into a sst writer/addsstable -- I didn't do as much benchmarking of this path, so yeah, I think we'd want to watch roachperf. I would expect backfills that race with a workload to maybe get slightly slower -- by design -- but those that are uncontended shouldn't be slowed much, if at all. |
b523db9 to
7a5e69d
Compare
rafiss
left a comment
There was a problem hiding this comment.
thanks for pushing this through!
if you're able to do it before merging, could you update the elastic=false variant of the schemachange/bulkingest test in pkg/cmd/roachtest/tests/schemachange.go so that it sets bulkio.index_backfill.elastic_control.enabled to false?
7a5e69d to
53beaf3
Compare
This change enables the bulkio.index_backfill.elastic_control.enabled cluster setting by default, integrating index backfill operations with elastic CPU control. This allows index backfill workloads to automatically throttle based on available CPU resources, improving cluster stability during schema changes. Release note (ops change): The bulkio.index_backfill.elastic_control.enabled cluster setting is now enabled by default, allowing index backfill operations to integrate with elastic CPU control and automatically throttle based on available resources. Epic: CRDB-48845.
53beaf3 to
8177267
Compare
done! TFTR! /trunk merge |
This change enables the bulkio.index_backfill.elastic_control.enabled cluster setting by default, integrating index backfill operations with elastic CPU control. This allows index backfill workloads to automatically throttle based on available CPU resources, improving cluster stability during schema changes.
Release note (ops change): The bulkio.index_backfill.elastic_control.enabled cluster setting is now enabled by default, allowing index backfill operations to integrate with elastic CPU control and automatically throttle based on available resources.
Epic: CRDB-48845.