Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[xCluster] Set enable_log_retention_by_op_idx to true by default #9662

Closed
rahuldesirazu opened this issue Aug 11, 2021 · 0 comments
Closed

[xCluster] Set enable_log_retention_by_op_idx to true by default #9662

rahuldesirazu opened this issue Aug 11, 2021 · 0 comments
Assignees
@rahuldesirazu
Labels
area/cdc Change Data Capture
Projects
xCluster replication

Comments

@rahuldesirazu
Copy link
Contributor

--enable_log_retention_by_op_idx controls whether we retain logs that haven't yet been replicated in an xCluster setup. Turn this to true by default and ensure we only trigger such retention when there are xCluster streams on a cluster.
@rahuldesirazu rahuldesirazu self-assigned this Aug 11, 2021
@rahuldesirazu rahuldesirazu added this to To do in xCluster replication via automation Aug 11, 2021
@bmatican bmatican mentioned this issue Aug 12, 2021
Closed
rahuldesirazu added a commit that referenced this issue Aug 20, 2021
@rahuldesirazu
[#9662][#9417] Set enable_log_retention_by_op_idx to true by default…
2933bf5 
… and bump update_metrics_interval_ms to 15000

Summary:
#9662:
Currently a user has to set enable_log_retention_by_op_idx to true for xCluster setups. We can default this flag to true, since we only change WAL retention policy for entries in the cdc_state table. So for setups with no xCluster enabled, changing this flag will have no effect.

#9417:
We currently set --update_metrics_interval_ms = 1000, which means we collect metrics every 1s. This operation opens the cdc_state table and loops through all tablets under-replication, which can be an expensive operation. Since we don't need such fine-grained metrics, we can change this to 15000 to collect every 15s.

Test Plan: Jenkins

Reviewers: nicolas, jhe

Reviewed By: jhe

Subscribers: ybase, bogdan

Differential Revision: https://phabricator.dev.yugabyte.com/D12638
rahuldesirazu added a commit that referenced this issue Aug 31, 2021
@rahuldesirazu
[Backport to 2.6][#9662][#9417] Set enable_log_retention_by_op_idx to…
06aa271 
… true by default and bump update_metrics_interval_ms to 15000

Summary:
#9662:
Currently a user has to set enable_log_retention_by_op_idx to true for xCluster setups. We can default this flag to true, since we only change WAL retention policy for entries in the cdc_state table. So for setups with no xCluster enabled, changing this flag will have no effect.

#9417:
We currently set --update_metrics_interval_ms = 1000, which means we collect metrics every 1s. This operation opens the cdc_state table and loops through all tablets under-replication, which can be an expensive operation. Since we don't need such fine-grained metrics, we can change this to 15000 to collect every 15s.

Original Commit : D12638 /2933bf506fbf7b6ac8fe6d6715fa4c03b8dcd550

Test Plan: Jenkins: rebase: 2.6

Reviewers: nicolas, jhe

Reviewed By: jhe

Subscribers: ybase, bogdan

Differential Revision: https://phabricator.dev.yugabyte.com/D12804
@nspiegelberg nspiegelberg mentioned this issue Sep 9, 2021
Open
xCluster replication automation moved this from To do to Done Sep 9, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rahuldesirazu rahuldesirazu

Labels
area/cdc Change Data Capture
Projects
xCluster replication
Done
Milestone
No milestone
Development

No branches or pull requests
2 participants
@rahuldesirazu @nspiegelberg