changefeedccl: use new bulk oracle for planning #119777
Assignees
Labels
A-cdc
Change Data Capture
C-enhancement
Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)
T-cdc
Comments
rharding6373
added
C-enhancement
![blathers[crl]](https://avatars.githubusercontent.com/in/59700?s=40&u=373d9101a12edc3847d2fb6ae78e3adcfb4ca41e&v=4){kind=avatar}
|
cc @cockroachdb/cdc |
craig bot
pushed a commit
that referenced
this issue
Mar 21, 2024
120077: changefeedccl: use new bulk oracle for changefeed planning r=rharding6373 a=rharding6373 This change uses the BulkOracle by default as part of changefeed planning, instead of the bin packing oracle. This will allow changefeeds to have plans that randomly assign spans to any replica, including followers if enabled, following locality filter constraints. A new cluster setting, `changefeed.balanced_distribution.enabled`, protects this change. When enabled (by default), changefeeds will use the new BulkOracle for planning. If disabled, changefeeds will use the previous bin packing oracle. Epic: none Fixes: #119777 Fixes: #114611 Release note (enterprise change): Changefeeds now use the BulkOracle for planning, which distributes work evenly across all replica in the locality filter, including followers if enabled. This is enabled by default with the cluster setting `changefeed.balanced_distribution.enabled`. If disabled, changefeed planning reverts to its previous bin packing oracle. Co-authored-by: rharding6373 <harding@cockroachlabs.com>
rharding6373
added a commit
to rharding6373/cockroach
that referenced
this issue
May 31, 2024
This change uses the BulkOracle by default as part of changefeed planning, instead of the bin packing oracle. This will allow changefeeds to have plans that randomly assign spans to any replica, including followers if enabled, following locality filter constraints. A new cluster setting, `changefeed.balanced_distribution.enabled`, protects this change. When enabled (by default), changefeeds will use the new BulkOracle for planning. If disabled, changefeeds will use the previous bin packing oracle. Epic: none Fixes: cockroachdb#119777 Fixes: cockroachdb#114611 Release note (enterprise change): Changefeeds now use the BulkOracle for planning, which distributes work evenly across all replica in the locality filter, including followers if enabled. This is enabled by default with the cluster setting `changefeed.balanced_distribution.enabled`. If disabled, changefeed planning reverts to its previous bin packing oracle.
rharding6373
added a commit
to rharding6373/cockroach
that referenced
this issue
May 31, 2024
This change uses the BulkOracle by default as part of changefeed planning, instead of the bin packing oracle. This will allow changefeeds to have plans that randomly assign spans to any replica, including followers if enabled, following locality filter constraints. A new cluster setting, `changefeed.balanced_distribution.enabled`, protects this change. When enabled (by default), changefeeds will use the new BulkOracle for planning. If disabled, changefeeds will use the previous bin packing oracle. Epic: none Fixes: cockroachdb#119777 Fixes: cockroachdb#114611 Release note (enterprise change): Changefeeds now use the BulkOracle for planning, which distributes work evenly across all replica in the locality filter, including followers if enabled. This is enabled by default with the cluster setting `changefeed.balanced_distribution.enabled`. If disabled, changefeed planning reverts to its previous bin packing oracle.
rharding6373
added a commit
to rharding6373/cockroach
that referenced
this issue
May 31, 2024
This change uses the BulkOracle by default as part of changefeed planning, instead of the bin packing oracle. This will allow changefeeds to have plans that randomly assign spans to any replica, including followers if enabled, following locality filter constraints. A new cluster setting, `changefeed.balanced_distribution.enabled`, protects this change. When enabled (by default), changefeeds will use the new BulkOracle for planning. If disabled, changefeeds will use the previous bin packing oracle. Epic: none Fixes: cockroachdb#119777 Fixes: cockroachdb#114611 Release note (enterprise change): Changefeeds now use the BulkOracle for planning, which distributes work evenly across all replica in the locality filter, including followers if enabled. This is enabled by default with the cluster setting `changefeed.balanced_distribution.enabled`. If disabled, changefeed planning reverts to its previous bin packing oracle.
rharding6373
added a commit
to rharding6373/cockroach
that referenced
this issue
May 31, 2024
This change uses the BulkOracle by default as part of changefeed planning, instead of the bin packing oracle. This will allow changefeeds to have plans that randomly assign spans to any replica, including followers if enabled, following locality filter constraints. A new cluster setting, `changefeed.balanced_distribution.enabled`, protects this change. When enabled (by default), changefeeds will use the new BulkOracle for planning. If disabled, changefeeds will use the previous bin packing oracle. Epic: none Fixes: cockroachdb#119777 Fixes: cockroachdb#114611 Release note (enterprise change): Changefeeds now use the BulkOracle for planning, which distributes work evenly across all replica in the locality filter, including followers if enabled. This is enabled by default with the cluster setting `changefeed.balanced_distribution.enabled`. If disabled, changefeed planning reverts to its previous bin packing oracle.
rharding6373
added a commit
to rharding6373/cockroach
that referenced
this issue
Jun 3, 2024
This change uses the BulkOracle by default as part of changefeed planning, instead of the bin packing oracle. This will allow changefeeds to have plans that randomly assign spans to any replica, including followers if enabled, following locality filter constraints. A new cluster setting, `changefeed.balanced_distribution.enabled`, protects this change. When enabled (by default), changefeeds will use the new BulkOracle for planning. If disabled, changefeeds will use the previous bin packing oracle. Epic: none Fixes: cockroachdb#119777 Fixes: cockroachdb#114611 Release note (enterprise change): Changefeeds now use the BulkOracle for planning, which distributes work evenly across all replica in the locality filter, including followers if enabled. This is enabled by default with the cluster setting `changefeed.balanced_distribution.enabled`. If disabled, changefeed planning reverts to its previous bin packing oracle.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
A new bulk oracle was added in #114710 to more evenly distribute spans across available nodes. This would be advantageous to use for the changefeed use case, too.
This could be an alternative or supplement to work to improve changefeed planning heuristics in #113898.
Jira issue: CRDB-36307
The text was updated successfully, but these errors were encountered: