Skip to content
This repository has been archived by the owner on Jun 23, 2022. It is now read-only.

feat(dup): add interface mutation_duplicator & duplication procedure #304

Merged
merged 5 commits into from
Sep 6, 2019

Conversation

neverchanje
Copy link
Contributor

@neverchanje neverchanje commented Aug 29, 2019

This PR added interface mutation_duplicator and pipeline of duplication procedure.

mutation_duplicator

This is the interface of data "sender" to be extensible for multiple endpoint types.
For example, if the remote endpoint is a pegasus cluster, the pegasus_mutation_duplicator will
transfer client RPCs to the remote pegasus.
If there's a further requirement to duplicate data to HBase/MySQL/etc., we can implement this interface.

Duplication Pipeline

The other part of this PR is simple. In each round of duplication, a batch of mutations is loaded
from private-log, and is sent via mutation_duplicator to remote endpoint.

So the pipeline can be declared like the following (in bash convention):

while true
do
  load_from_private_log | ship_mutations
done

Each procedure in our implementation is declared as a class:

  • load_from_private_log, run in thread pool, REPLICATION_LONG
  • ship_mutations, run in thread pool, REPLICATION

In our real implementation, we make this design slightly complicated:

while true
do
  load_mutations | ship_mutations
done

# load_mutations will fork a new task to load from private-log, aka `load_from_private_log`
  • load_mutations runs in thread pool REPLICATION, because we need to obtain the latest information of this replica. Leave to future refactoring if we find this unnecessary and we can delete this class.

replica_duplicator

replica_duplicator is the basement of this pipeline, so it extends pipeline::base.

thread_pool(LPC_REPLICATION_LOW).task_tracker(tracker()).thread_hash(get_gpid().thread_hash());

It means that the pipeline is running in the environment of:

  • LPC_REPLICATION_LOW: in low priority of thread pool REPLICATION.
  • using the task tracker of replica, which means when the replica destructs, the pipeline will be detroyed.
  • using the thread hash of this replica.

qinzuoyan
qinzuoyan previously approved these changes Sep 4, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants