Snapshot Establishment - tserver changes #19679
Assignees
Labels
jira-originated
kind/new-feature
This is a request for a completely new feature
priority/low
Low priority
Comments
yugabyte-ci
added
jira-originated
kind/new-feature
yugabyte-ci
changed the title
asrinivasanyb
added a commit
that referenced
this issue
Dec 5, 2023
…ocated table support Summary: This is a part of the CDC Consistent Snapshot feature (GH #18508). This diff includes the changes related to snapshot and streaming changes consumption for consistent snapshot streams. This diff also includes support for colocated tables in the context of consistent snapshot stream. The master side changes are covered by GH #19678 The tserver side changes are covered by GH #19679 As such, there are no major changes to the consumption side. Snapshot records and Change records should be exhaustive and mutually exclusive. Mutual exclusion in this context means that none of the DMLs prior to the snapshot should be considered as changes. To ensure this, The safe_hybrid_time filter will be set to the maximum of the snapshot_time and the safe_hybrid_time that is passed in as part of the request. This is to filter out WAL records that have a commit_time lower than or equal to the snapshot_time. The support for colocated tables involves inserting the colocated table snapshot row as part of the create stream workflow. UPGRADE/ROLLBACK SAFETY: Upgrade safe Rollback safe PB changes 1) yb::master::CDCStreamInfoPB - addition of 2 optional fields. This is used in an RPC response 2) yb::master::CDCSDKStreamEntryPB - addition of 1 optional field - This is a disk format change This feature is currently disabled. Changes are guarded by the TEST flag yb_enable_cdc_consistent_snapshot_streams which is set to false by default. This same flag will be redefined as an autoflag (LocalPersisted) once all the support for this feature is in place. Jira: DB-8476 Test Plan: New Unit Tests - Ported 18 snapshot related tests to use consistent snapshot streams. ./yb_build.sh --cxx-test integration-tests_cdcsdk_consistent_snapshot-test ./yb_build.sh --cxx-test integration-tests_cdcsdk_consistent_snapshot-test --gtest_filter CDCSDKConsistentSnapshotTest.TestConsistentSnapshotAcrossMultipleTables Existing CDCSDK tests Reviewers: skumar, stiwary, vkushwaha, xCluster, hsunder Reviewed By: stiwary Subscribers: ycdcxcluster, ybase, bogdan Differential Revision: https://phorge.dev.yugabyte.com/D30496
asrinivasanyb
added a commit
that referenced
this issue
Dec 14, 2023
…server changes Summary: **Backport Description** There was one simple merge conflict related to Flag declaration in cdcsdk_ysql_test_base.h **Original Description** Original commit: 11a0bd5 / D30088 This is a part of the CDC Consistent Snapshot feature (GH #18508). This diff includes the changes on the tserver side to the create stream workflow. The master side changes are covered by GH #19678 The key task to accomplish here is to ensure that from the point of stream creation, the sources of information are all preserved until the snapshot records and the changes are consumed by the client. For snapshot records, the main sources of information are the RegularDb and IntentsDb. Specifically, the contents of the tablets (the set of records), as of the snapshot time should be available. That is, the versions of the K/V as of snapshot time should be available. This is referred to as history retention. This information could be lost when compactions occur. For changes, the main sources of information are the WAL and IntentsDb. The information that is required from the WAL could be lost when WAL GC happens. The information that is required from the IntentsDb could be lost when Intents are deleted after applying to the RegularDb Thus, setting retention barriers on WAL/Intents/RegularDB is the main task accomplished here. The key idea is to perform these preparatory retention steps as part of the Apply of a RAFT operation(ChangeMetadataOperation). This is to ensure that the followers will also be able to deliver the snapshot and will not miss any changes in the event that they have to take over as the leader. In addition, each tablet leader also responds with an OpId from where the scanning of the WAL of that tablet for the change records can safely commence. More details of the algorithm and analysis can be found in the [[ https://docs.google.com/document/d/1kSAWAbctjw6q58zKrJav0GKjWo5QTDfeJvkoHKP-DXE/edit | Design Spec ]]. UPGRADE/ROLLBACK SAFETY: Upgrade safe Rollback safe This feature is currently disabled. Changes are guarded by the TEST flag yb_enable_cdc_consistent_snapshot_streams which is set to false by default. This same flag will be redefined as an autoflag (LocalPersisted) once all the support for this feature is in place. Jira: DB-8475 Test Plan: New Unit Tests - ./yb_build.sh --cxx-test integration-tests_cdcsdk_snapshot-test --gtest_filter CDCSDKYsqlTest.TestCSStreamSnapshotEstablishment ./yb_build.sh --cxx-test integration-tests_cdcsdk_snapshot-test --gtest_filter CDCSDKYsqlTest.TestTwoCSStream Existing CDCSDK tests Existing ALTER TABLE tests Reviewers: sergei, skumar, hsunder Reviewed By: skumar Subscribers: ybase, ycdcxcluster, bogdan, hsunder Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D31066
asrinivasanyb
added a commit
that referenced
this issue
Dec 14, 2023
…changes and colocated table support Summary: **Backport Description** The merge was clean. **Original Description** Original commit: e013c72 / D30496 This is a part of the CDC Consistent Snapshot feature (GH #18508). This diff includes the changes related to snapshot and streaming changes consumption for consistent snapshot streams. This diff also includes support for colocated tables in the context of consistent snapshot stream. The master side changes are covered by GH #19678 The tserver side changes are covered by GH #19679 As such, there are no major changes to the consumption side. Snapshot records and Change records should be exhaustive and mutually exclusive. Mutual exclusion in this context means that none of the DMLs prior to the snapshot should be considered as changes. To ensure this, The safe_hybrid_time filter will be set to the maximum of the snapshot_time and the safe_hybrid_time that is passed in as part of the request. This is to filter out WAL records that have a commit_time lower than or equal to the snapshot_time. The support for colocated tables involves inserting the colocated table snapshot row as part of the create stream workflow. UPGRADE/ROLLBACK SAFETY: Upgrade safe Rollback safe PB changes 1) yb::master::CDCStreamInfoPB - addition of 2 optional fields. This is used in an RPC response 2) yb::master::CDCSDKStreamEntryPB - addition of 1 optional field - This is a disk format change This feature is currently disabled. Changes are guarded by the TEST flag yb_enable_cdc_consistent_snapshot_streams which is set to false by default. This same flag will be redefined as an autoflag (LocalPersisted) once all the support for this feature is in place. Jira: DB-8476 Test Plan: New Unit Tests - Ported 18 snapshot related tests to use consistent snapshot streams. ./yb_build.sh --cxx-test integration-tests_cdcsdk_consistent_snapshot-test ./yb_build.sh --cxx-test integration-tests_cdcsdk_consistent_snapshot-test --gtest_filter CDCSDKConsistentSnapshotTest.TestConsistentSnapshotAcrossMultipleTables Existing CDCSDK tests Reviewers: skumar, stiwary, vkushwaha, xCluster, hsunder Reviewed By: skumar Subscribers: bogdan, ybase, ycdcxcluster Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D31070
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Jira Link: DB-8475
The text was updated successfully, but these errors were encountered: