[CDC] Setup CDC Consumer for reading changes for set of assigned tablets #1481
Assignees
Labels
Projects
Comments
rahuldesirazu
added a commit
that referenced
this issue
Aug 29, 2019
Summary: This code implements 3 components: 1. Master level changes. Add a new field to the sys catalog with the `consumer -> producer` tablet mapping. Create a `RegisterSubscriber(producer_table, consumer_table)` RPC which adds new entries to the mapping. Add a the new mapping to the heartbeat response if it changed since the last hb from that tserver. NOTE: There is currently no support for removing subscribers or for non 1 : 1 mapping between consumer and producer tablets (i.e. consumer and producer tables must have same number of tablets). 2. CDCConsumer level changes. Create a CDCConsumer object that lives on the tserver. When the tserver processes a heartbeat with a CDC mapping, go through the mapping and kick off getting changes if the tserver is leader for that tablet. The flow is `RegisterConsumerTablet -> Poll -> HandlePoll -> Apply -> HandleApply -> Poll` in a loop as long as the peer is leader. Again note there is no support for removing subscribers yet. 3. Raft level changes. To ensure that only the tablet leader is responsible for polling, modify Raft to include become leader and become follower clbks. The become leader clbk is responsible for kicking off RegisterConsumerTablet if CDC is enabled, while the become follower clbk sets a marker to stop the poll process. To rebase on top of this diff, do the following: 1. Modify `TwoDCOutputClient::ApplyChanges` with the 2DC implementation for applying polling changes. This is a virtual method that will have 2dc, kafka, and elastic implementations. Most likely, you will need to include a HandleApplyChanges callback method for the async call, which is responsible for calling `apply_changes_clbk` with the status and last applied op id. 2. Modify the CDCServiceProxy to correctly get the rpc addr of the producer universe. Right now, this diff assumes the producer and consumer tables live on the same universe, so a local call is used. 3. Make sure `CDCTest::BasicWriteRead` passes. Super simple non-txn test that writes entries to producer table and makes sure same entries appear on consumer table. Test Plan: `TwoDCTest::PollWithProducerRestart TwoDCTest::PollWithConsumerRestart` Reviewers: bogdan, sergei, neha Reviewed By: neha Subscribers: ybase, bogdan, neha Differential Revision: https://phabricator.dev.yugabyte.com/D6913
d-uspenskiy
pushed a commit
to d-uspenskiy/yugabyte-db
that referenced
this issue
Sep 5, 2019
Summary: This code implements 3 components: 1. Master level changes. Add a new field to the sys catalog with the `consumer -> producer` tablet mapping. Create a `RegisterSubscriber(producer_table, consumer_table)` RPC which adds new entries to the mapping. Add a the new mapping to the heartbeat response if it changed since the last hb from that tserver. NOTE: There is currently no support for removing subscribers or for non 1 : 1 mapping between consumer and producer tablets (i.e. consumer and producer tables must have same number of tablets). 2. CDCConsumer level changes. Create a CDCConsumer object that lives on the tserver. When the tserver processes a heartbeat with a CDC mapping, go through the mapping and kick off getting changes if the tserver is leader for that tablet. The flow is `RegisterConsumerTablet -> Poll -> HandlePoll -> Apply -> HandleApply -> Poll` in a loop as long as the peer is leader. Again note there is no support for removing subscribers yet. 3. Raft level changes. To ensure that only the tablet leader is responsible for polling, modify Raft to include become leader and become follower clbks. The become leader clbk is responsible for kicking off RegisterConsumerTablet if CDC is enabled, while the become follower clbk sets a marker to stop the poll process. To rebase on top of this diff, do the following: 1. Modify `TwoDCOutputClient::ApplyChanges` with the 2DC implementation for applying polling changes. This is a virtual method that will have 2dc, kafka, and elastic implementations. Most likely, you will need to include a HandleApplyChanges callback method for the async call, which is responsible for calling `apply_changes_clbk` with the status and last applied op id. 2. Modify the CDCServiceProxy to correctly get the rpc addr of the producer universe. Right now, this diff assumes the producer and consumer tables live on the same universe, so a local call is used. 3. Make sure `CDCTest::BasicWriteRead` passes. Super simple non-txn test that writes entries to producer table and makes sure same entries appear on consumer table. Test Plan: `TwoDCTest::PollWithProducerRestart TwoDCTest::PollWithConsumerRestart` Reviewers: bogdan, sergei, neha Reviewed By: neha Subscribers: ybase, bogdan, neha Differential Revision: https://phabricator.dev.yugabyte.com/D6913
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
No description provided.
The text was updated successfully, but these errors were encountered: