[CDCSDK] Support changing the value of publication refresh interval and change precision of the refresh interval flag #21796
Assignees
Labels
area/cdcsdk
CDC SDK
jira-originated
kind/enhancement
This is an enhancement of an existing feature
priority/high
High Priority
Comments
yugabyte-ci
added
area/cdcsdk
yugabyte-ci
changed the title
Sumukh-Phalgaonkar
added a commit
that referenced
this issue
Apr 30, 2024
…alue Summary: In this diff, support has been added to allow changing the value of publication refresh interval via the flag `cdcsdk_publication_list_refresh_interval_secs`. Inorder to protect LSN determinism, the values of publication refresh times will be persisted until a suitable acknowledgement is received. For this purpose a new key has been added to the data map in cdc_state called `pub_refresh_times`. It will contain coma separated values of the publication refresh times which have been popped from the priority queue but not yet acknowledged. In the GetConsistentChanges, whenever the tablet queue for publication_refresh_records becomes empty, the new entry added to the tablet queue will also be added to `pub_refresh_times` list and persisted in the state table. This will ensure that before shipping any LSN with commit time greater than the last_pub_refresh_time, we persist the next pub_refresh_time. When an acknowledgement reaches virtual WAL, in `UpdateAndPersistLSN()`, the `pub_refresh_times` list will be trimmed, so that it contains only those values which are strictly greater than the acknowledged publication refresh time. The field `last_pub_refresh_time` in the state table slot entry will hold the latest acknowledged publication refresh time. In this diff the precision of the flag `cdcsdk_publication_list_refresh_interval` has also been changed from microseconds to seconds. This change is done to improve the usability of the flag. For the test purposes the flags `TEST_cdcsdk_use_microseconds_refresh_interval` and `TEST_cdcsdk_publication_list_refresh_interval_micros` can be used to set the refresh interval in microseconds. Jira: DB-10688 Test Plan: Jenkins: urgent Jenkins: test regex: .*CDCSDKConsumptionConsistentChangesTest.* ./yb_build.sh --cxx-test integration-tests_cdcsdk_consumption_consistent_changes-test --gtest_filter CDCSDKConsumptionConsistentChangesTest.TestChangingPublicationRefreshInterval ./yb_build.sh --cxx-test integration-tests_cdcsdk_consumption_consistent_changes-test --gtest_filter CDCSDKConsumptionConsistentChangesTest.TestLSNDeterminismWithChangingPubRefreshInterval Reviewers: skumar, asrinivasan, stiwary, siddharth.shah Reviewed By: asrinivasan Subscribers: ybase, ycdcxcluster Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D34460
Sumukh-Phalgaonkar
added a commit
that referenced
this issue
Apr 30, 2024
…h Interval Flag value Summary: Original commit: 90cb25b / D34460 In this diff, support has been added to allow changing the value of publication refresh interval via the flag `cdcsdk_publication_list_refresh_interval_secs`. Inorder to protect LSN determinism, the values of publication refresh times will be persisted until a suitable acknowledgement is received. For this purpose a new key has been added to the data map in cdc_state called `pub_refresh_times`. It will contain coma separated values of the publication refresh times which have been popped from the priority queue but not yet acknowledged. In the GetConsistentChanges, whenever the tablet queue for publication_refresh_records becomes empty, the new entry added to the tablet queue will also be added to `pub_refresh_times` list and persisted in the state table. This will ensure that before shipping any LSN with commit time greater than the last_pub_refresh_time, we persist the next pub_refresh_time. When an acknowledgement reaches virtual WAL, in `UpdateAndPersistLSN()`, the `pub_refresh_times` list will be trimmed, so that it contains only those values which are strictly greater than the acknowledged publication refresh time. The field `last_pub_refresh_time` in the state table slot entry will hold the latest acknowledged publication refresh time. In this diff the precision of the flag `cdcsdk_publication_list_refresh_interval` has also been changed from microseconds to seconds. This change is done to improve the usability of the flag. For the test purposes the flags `TEST_cdcsdk_use_microseconds_refresh_interval` and `TEST_cdcsdk_publication_list_refresh_interval_micros` can be used to set the refresh interval in microseconds. ######Backport Description No merge conflicts were encountered. Jira: DB-10688 Test Plan: Jenkins: urgent Jenkins: test regex: .*CDCSDKConsumptionConsistentChangesTest.* ./yb_build.sh --cxx-test integration-tests_cdcsdk_consumption_consistent_changes-test --gtest_filter CDCSDKConsumptionConsistentChangesTest.TestChangingPublicationRefreshInterval ./yb_build.sh --cxx-test integration-tests_cdcsdk_consumption_consistent_changes-test --gtest_filter CDCSDKConsumptionConsistentChangesTest.TestLSNDeterminismWithChangingPubRefreshInterval Reviewers: skumar, asrinivasan, stiwary, siddharth.shah Reviewed By: siddharth.shah Subscribers: ycdcxcluster, ybase Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D34628
svarnau
pushed a commit
that referenced
this issue
May 25, 2024
…alue Summary: In this diff, support has been added to allow changing the value of publication refresh interval via the flag `cdcsdk_publication_list_refresh_interval_secs`. Inorder to protect LSN determinism, the values of publication refresh times will be persisted until a suitable acknowledgement is received. For this purpose a new key has been added to the data map in cdc_state called `pub_refresh_times`. It will contain coma separated values of the publication refresh times which have been popped from the priority queue but not yet acknowledged. In the GetConsistentChanges, whenever the tablet queue for publication_refresh_records becomes empty, the new entry added to the tablet queue will also be added to `pub_refresh_times` list and persisted in the state table. This will ensure that before shipping any LSN with commit time greater than the last_pub_refresh_time, we persist the next pub_refresh_time. When an acknowledgement reaches virtual WAL, in `UpdateAndPersistLSN()`, the `pub_refresh_times` list will be trimmed, so that it contains only those values which are strictly greater than the acknowledged publication refresh time. The field `last_pub_refresh_time` in the state table slot entry will hold the latest acknowledged publication refresh time. In this diff the precision of the flag `cdcsdk_publication_list_refresh_interval` has also been changed from microseconds to seconds. This change is done to improve the usability of the flag. For the test purposes the flags `TEST_cdcsdk_use_microseconds_refresh_interval` and `TEST_cdcsdk_publication_list_refresh_interval_micros` can be used to set the refresh interval in microseconds. Jira: DB-10688 Test Plan: Jenkins: urgent Jenkins: test regex: .*CDCSDKConsumptionConsistentChangesTest.* ./yb_build.sh --cxx-test integration-tests_cdcsdk_consumption_consistent_changes-test --gtest_filter CDCSDKConsumptionConsistentChangesTest.TestChangingPublicationRefreshInterval ./yb_build.sh --cxx-test integration-tests_cdcsdk_consumption_consistent_changes-test --gtest_filter CDCSDKConsumptionConsistentChangesTest.TestLSNDeterminismWithChangingPubRefreshInterval Reviewers: skumar, asrinivasan, stiwary, siddharth.shah Reviewed By: asrinivasan Subscribers: ybase, ycdcxcluster Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D34460
ZhenYongFan
pushed a commit
to ZhenYongFan/yugabyte-db
that referenced
this issue
Jun 15, 2024
…n Refresh Interval Flag value Summary: Original commit: 90cb25b / D34460 In this diff, support has been added to allow changing the value of publication refresh interval via the flag `cdcsdk_publication_list_refresh_interval_secs`. Inorder to protect LSN determinism, the values of publication refresh times will be persisted until a suitable acknowledgement is received. For this purpose a new key has been added to the data map in cdc_state called `pub_refresh_times`. It will contain coma separated values of the publication refresh times which have been popped from the priority queue but not yet acknowledged. In the GetConsistentChanges, whenever the tablet queue for publication_refresh_records becomes empty, the new entry added to the tablet queue will also be added to `pub_refresh_times` list and persisted in the state table. This will ensure that before shipping any LSN with commit time greater than the last_pub_refresh_time, we persist the next pub_refresh_time. When an acknowledgement reaches virtual WAL, in `UpdateAndPersistLSN()`, the `pub_refresh_times` list will be trimmed, so that it contains only those values which are strictly greater than the acknowledged publication refresh time. The field `last_pub_refresh_time` in the state table slot entry will hold the latest acknowledged publication refresh time. In this diff the precision of the flag `cdcsdk_publication_list_refresh_interval` has also been changed from microseconds to seconds. This change is done to improve the usability of the flag. For the test purposes the flags `TEST_cdcsdk_use_microseconds_refresh_interval` and `TEST_cdcsdk_publication_list_refresh_interval_micros` can be used to set the refresh interval in microseconds. ######Backport Description No merge conflicts were encountered. Jira: DB-10688 Test Plan: Jenkins: urgent Jenkins: test regex: .*CDCSDKConsumptionConsistentChangesTest.* ./yb_build.sh --cxx-test integration-tests_cdcsdk_consumption_consistent_changes-test --gtest_filter CDCSDKConsumptionConsistentChangesTest.TestChangingPublicationRefreshInterval ./yb_build.sh --cxx-test integration-tests_cdcsdk_consumption_consistent_changes-test --gtest_filter CDCSDKConsumptionConsistentChangesTest.TestLSNDeterminismWithChangingPubRefreshInterval Reviewers: skumar, asrinivasan, stiwary, siddharth.shah Reviewed By: siddharth.shah Subscribers: ycdcxcluster, ybase Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D34628
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Jira Link: DB-10688
The text was updated successfully, but these errors were encountered: