Handle deletion of cdc_state entry for slot when stream is deleted #21204
Assignees
Labels
2024.1_blocker
2024.1.0_blocker
area/cdc
Change Data Capture
jira-originated
kind/new-feature
This is a request for a completely new feature
priority/medium
Medium priority issue
Comments
yugabyte-ci
added
area/cdc
yugabyte-ci
added
priority/high
yugabyte-ci
added
priority/medium
Sumukh-Phalgaonkar
added a commit
that referenced
this issue
May 10, 2024
Summary: This diff adds support to delete the cdc_state table entry for the slot, when the stream corresponding to the slot is deleted. The master background task sets the checkpoint to max for all the state table entries corresponding to the stream that is being deleted. The slot row entry is no exception to this. We use this max checkpoint information to find out the slot entries that need to be deleted. The slot entry is used for setting of `cdc_sdk_safe_time` for the other entries for that stream, hence while deleting we need to ensure that the slot entry is deleted only after all the entries with valid tablet_id for that stream are deleted. We accomplish this by deleting the entries with valid tablet_id first and then deleting slot entry in the next pass of `UpdatePeersAndMetrics`. Jira: DB-10134 Test Plan: Jenkins: .*CDCSDK.* ./yb_build.sh --cxx-test integration-tests_cdcsdk_consumption_consistent_changes-test --gtest_filter CDCSDKConsumptionConsistentChangesTest.TestSlotRowDeletionWithSingleStream ./yb_build.sh --cxx-test integration-tests_cdcsdk_consumption_consistent_changes-test --gtest_filter CDCSDKConsumptionConsistentChangesTest.TestSlotRowDeletionWithMultipleStreams Reviewers: asrinivasan, skumar, stiwary, siddharth.shah Reviewed By: asrinivasan Subscribers: ycdcxcluster Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D34828
Sumukh-Phalgaonkar
added a commit
that referenced
this issue
May 14, 2024
…from cdc_state table Summary: Original commit: bc46258 / D34828 This diff adds support to delete the cdc_state table entry for the slot, when the stream corresponding to the slot is deleted. The master background task sets the checkpoint to max for all the state table entries corresponding to the stream that is being deleted. The slot row entry is no exception to this. We use this max checkpoint information to find out the slot entries that need to be deleted. The slot entry is used for setting of `cdc_sdk_safe_time` for the other entries for that stream, hence while deleting we need to ensure that the slot entry is deleted only after all the entries with valid tablet_id for that stream are deleted. We accomplish this by deleting the entries with valid tablet_id first and then deleting slot entry in the next pass of `UpdatePeersAndMetrics`. #####Backport Description No merge conflicts were encountered. Jira: DB-10134 Test Plan: Jenkins: .*CDCSDK.* ./yb_build.sh --cxx-test integration-tests_cdcsdk_consumption_consistent_changes-test --gtest_filter CDCSDKConsumptionConsistentChangesTest.TestSlotRowDeletionWithSingleStream ./yb_build.sh --cxx-test integration-tests_cdcsdk_consumption_consistent_changes-test --gtest_filter CDCSDKConsumptionConsistentChangesTest.TestSlotRowDeletionWithMultipleStreams Reviewers: asrinivasan, skumar, stiwary, siddharth.shah Reviewed By: asrinivasan Subscribers: ycdcxcluster Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D34948
Sumukh-Phalgaonkar
added a commit
that referenced
this issue
May 14, 2024
…om cdc_state table Summary: Original commit: bc46258 / D34828 This diff adds support to delete the cdc_state table entry for the slot, when the stream corresponding to the slot is deleted. The master background task sets the checkpoint to max for all the state table entries corresponding to the stream that is being deleted. The slot row entry is no exception to this. We use this max checkpoint information to find out the slot entries that need to be deleted. The slot entry is used for setting of `cdc_sdk_safe_time` for the other entries for that stream, hence while deleting we need to ensure that the slot entry is deleted only after all the entries with valid tablet_id for that stream are deleted. We accomplish this by deleting the entries with valid tablet_id first and then deleting slot entry in the next pass of `UpdatePeersAndMetrics`. #####Backport Description No merge conflicts were encountered. Jira: DB-10134 Test Plan: Jenkins: .*CDCSDK.* ./yb_build.sh --cxx-test integration-tests_cdcsdk_consumption_consistent_changes-test --gtest_filter CDCSDKConsumptionConsistentChangesTest.TestSlotRowDeletionWithSingleStream ./yb_build.sh --cxx-test integration-tests_cdcsdk_consumption_consistent_changes-test --gtest_filter CDCSDKConsumptionConsistentChangesTest.TestSlotRowDeletionWithMultipleStreams Reviewers: asrinivasan, skumar, stiwary, siddharth.shah Reviewed By: asrinivasan Subscribers: ycdcxcluster Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D34947
svarnau
pushed a commit
that referenced
this issue
May 25, 2024
Summary: This diff adds support to delete the cdc_state table entry for the slot, when the stream corresponding to the slot is deleted. The master background task sets the checkpoint to max for all the state table entries corresponding to the stream that is being deleted. The slot row entry is no exception to this. We use this max checkpoint information to find out the slot entries that need to be deleted. The slot entry is used for setting of `cdc_sdk_safe_time` for the other entries for that stream, hence while deleting we need to ensure that the slot entry is deleted only after all the entries with valid tablet_id for that stream are deleted. We accomplish this by deleting the entries with valid tablet_id first and then deleting slot entry in the next pass of `UpdatePeersAndMetrics`. Jira: DB-10134 Test Plan: Jenkins: .*CDCSDK.* ./yb_build.sh --cxx-test integration-tests_cdcsdk_consumption_consistent_changes-test --gtest_filter CDCSDKConsumptionConsistentChangesTest.TestSlotRowDeletionWithSingleStream ./yb_build.sh --cxx-test integration-tests_cdcsdk_consumption_consistent_changes-test --gtest_filter CDCSDKConsumptionConsistentChangesTest.TestSlotRowDeletionWithMultipleStreams Reviewers: asrinivasan, skumar, stiwary, siddharth.shah Reviewed By: asrinivasan Subscribers: ycdcxcluster Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D34828
svarnau
pushed a commit
that referenced
this issue
May 29, 2024
…om cdc_state table Summary: Original commit: bc46258 / D34828 This diff adds support to delete the cdc_state table entry for the slot, when the stream corresponding to the slot is deleted. The master background task sets the checkpoint to max for all the state table entries corresponding to the stream that is being deleted. The slot row entry is no exception to this. We use this max checkpoint information to find out the slot entries that need to be deleted. The slot entry is used for setting of `cdc_sdk_safe_time` for the other entries for that stream, hence while deleting we need to ensure that the slot entry is deleted only after all the entries with valid tablet_id for that stream are deleted. We accomplish this by deleting the entries with valid tablet_id first and then deleting slot entry in the next pass of `UpdatePeersAndMetrics`. #####Backport Description No merge conflicts were encountered. Jira: DB-10134 Test Plan: Jenkins: .*CDCSDK.* ./yb_build.sh --cxx-test integration-tests_cdcsdk_consumption_consistent_changes-test --gtest_filter CDCSDKConsumptionConsistentChangesTest.TestSlotRowDeletionWithSingleStream ./yb_build.sh --cxx-test integration-tests_cdcsdk_consumption_consistent_changes-test --gtest_filter CDCSDKConsumptionConsistentChangesTest.TestSlotRowDeletionWithMultipleStreams Reviewers: asrinivasan, skumar, stiwary, siddharth.shah Reviewed By: asrinivasan Subscribers: ycdcxcluster Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D34947
ZhenYongFan
pushed a commit
to ZhenYongFan/yugabyte-db
that referenced
this issue
Jun 15, 2024
…lot row from cdc_state table Summary: Original commit: bc46258 / D34828 This diff adds support to delete the cdc_state table entry for the slot, when the stream corresponding to the slot is deleted. The master background task sets the checkpoint to max for all the state table entries corresponding to the stream that is being deleted. The slot row entry is no exception to this. We use this max checkpoint information to find out the slot entries that need to be deleted. The slot entry is used for setting of `cdc_sdk_safe_time` for the other entries for that stream, hence while deleting we need to ensure that the slot entry is deleted only after all the entries with valid tablet_id for that stream are deleted. We accomplish this by deleting the entries with valid tablet_id first and then deleting slot entry in the next pass of `UpdatePeersAndMetrics`. #####Backport Description No merge conflicts were encountered. Jira: DB-10134 Test Plan: Jenkins: .*CDCSDK.* ./yb_build.sh --cxx-test integration-tests_cdcsdk_consumption_consistent_changes-test --gtest_filter CDCSDKConsumptionConsistentChangesTest.TestSlotRowDeletionWithSingleStream ./yb_build.sh --cxx-test integration-tests_cdcsdk_consumption_consistent_changes-test --gtest_filter CDCSDKConsumptionConsistentChangesTest.TestSlotRowDeletionWithMultipleStreams Reviewers: asrinivasan, skumar, stiwary, siddharth.shah Reviewed By: asrinivasan Subscribers: ycdcxcluster Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D34948
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Jira Link: DB-10134
The text was updated successfully, but these errors were encountered: