[ROCKETMQ-193] Develop rocketmq-redis-replicator component #29
Conversation
|
@leonchen83 Thanks for your contribution. BTW, have you test it in your local environment ? |
|
@vongosling |
|
I would like to see your PR format like this. The following is template you would like to conform :-) Follow this checklist to help us incorporate your contribution quickly and easily:
|
leonchen83
changed the title
leonchen83
changed the title
Design: Redis supplies an official replication mechanism , and slave communicates to master with RESP protocol, so a natural way to design the rocketmq-redis-replicator component is simulating itself as a slave, sending commands to master and receiving datas from master timely, and then resending to rocketmq broker. If you are not familiar with redis replication mechanism, please learn this section first [1]. After that, I will illustrate some key points ahead. 1. To make slave start from the point where it left off when it reconnects, slave and master should agree on a master runId and a replication offset. Slave acknowledges this offset to master periodically. In other words，slave may received duplicate commands. Along with, the rocketmq-redis-replicator component may send duplicate messages too. A good way to minimize the duplicate time window is reducing the "ack period" to a smaller one, such as 100ms. 2. If slave keeps offline for some time, it’s easy to use up backlog whose default value is just 1M, especially for a high-traffic redis instance. Unfortunatelly,if slave replication offset has already been covered in master backlog, a full synchronization will have to execute, which is unacceptable for rocketmq-redis-replicator component as a large number of messages will be sent out intensively. 3. When synchronizing from master fully, master will generate a new rdb file(the rdb file format [2])，and slave will receive this file,store in disk, and last apply to memory. This strategy makes slave reaches a consistent state with master as soon as possible, and hardly fail. For rocketmq-redis-replicator component, it’s also a good way to prevent synchronizing initial rdb file from failure in halfway. There already an open source project [3] which focuses on replicating redis data, and provides api to handle data received [4]. The principal thoughts are simulating itself as a slave , following official replication procedure, communicating with master by RESP, and acking master with replication offset. Base on this project to develop is a good idea, meanwhile some aspects should also be enhanced and considered more robust. Here is some points: [High Available] Keeping the replication component's high availability is not difficult but important, not only for providing an uninterruptible service. If component leaves off for some time, a unacceptable full synchronization may be triggered. It’s also easy to reach high availability, including adopting master/slave module, using zookeeper to coordinate and switch master/slave, storing data onto zookeeper to keep component stateless. [Data Loss] Generally, data loss should be tried best to avoid. The key point is that slave only acks replication offset to master after sending command to rocketmq broker successfully. [Data Stale] It also happened when slave reconnect. Consider case below: `time1` `time2` `time3` set k=a set k=b set k=c If slave left off at time3, but the latest replication offset reported to master is only at time1, when slave reconnected, it re-apply commands “set k=b… set k=c”. In a small time window, “k” will equal the stale “b” until “set k=c” command is applied. So the slave offline time shorter, the better. [Message Order] Redis uses single thread model to keep command execute in order, because of its high performance. Replicating data with a single thread in slave is also fine, as it is also totally memory operation. But sending all data to rocketmq in a global order is a good choose? Producer should have no performance issue, but consumer may not be able to consume messages in time, especially redis was in a high load. Hashing “KEY” to different rocketmq queue is a good strategy. Guarantee the same key operation route to a unique queue, to keep partial ordered, and the downstream consumer could consume messages concurrently. Of course, some dependency “KEY”s may need hash to a unique queue too. We should supply configuration or api to support this individuation. [Transaction] Redis supports simple transaction. A transaction starts with a “MULTI” command, and redis buffers latter commands and execute them until receiving a “EXEC” command. But if one of the buffered commands executes fail, the preceding executed commands won’t roll back, and the latter commands will be discarded. So redis transaction could not guarantee atomic. In rocketmq, it’s also impossible to gather multi messages consume operation into a transaction. But the rocketmq-redis-replicator component will just receive multi commands after redis server get a “EXEC” command. From this aspect, the “transaction semantic doesn’t strengthen or weaken after this component resend messages. [Avoid component switched to master] In Sentinel or Redis Cluster, master crash could be detected by some mechanism, and one slave will be switched to master. Master has all information about its slaves, and the candidate slave is picked up automatically. Obviously, rocketmq-redis-replicator component have no ability to undertake master role. Configure this component to “read only” slave is a good way to avoid the component switch to master. [Support Redis Cluster] No matter Redis Cluster, or previous Partition, a slave only keeps track of one master. So the replication mechanism won’t change in single-node redis instance or a redis cluster. [1] : https://redis.io/topics/replication [2]: https://github.com/sripathikrishnan/redis-rdb-tools/wiki/Redis-RDB-Dump-File-Format [3]: https://github.com/leonchen83/redis-replicator [4]: https://github.com/leonchen83/redis-replicator#31-replication-via-socket
|
Write necessary unit-test to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in test module. Do you have test module in this replicator ? |
|
any reference or example for step 4 in other sub-project ? |
|
Consider this is the first commit for this project, I would like to recommend creating one module naming test for integration test :-) |
|
I recommend using travis ci to do integration test like other open source project. |
|
@vongosling I already added an integration-test to rocketmq-redis-replicator base on travis-ci. |
|
@vongosling you will see the result of integration-test at incubator-rocketmq-externals |
|
Hi, @leonchen83 don't forget to submit the Apache Individual Contributor License Agreement, since this is a large PR. |
|
And we need a detailed README to quick start with the projects~ 😄 |
remove unused boost file update README.md update readme.md update readme.md update readme add gitkeep to bin update readme.md update apache license info remove static dependency update readme.md add jsoncpp and libevent install discription udpate makefile update jsoncpp version unify dependency update readme support cmake instead alog by boost::log For cmake, remove alog remove alog comment [add rocketmq-cpp code #27] add cmake to readme use rocketmq namespace, update json version to 0.10.6 update cmakelist update json version to 0.10.6 remove makefile add default search path for jsoncpp and libevent Update README.md fix could not find jsoncpp issue support build on new g++ version Remove built .o files
|
@zhouxinyu ICLA already signed |
|
@zhouxinyu README.md added |
|
@leonchen83 Could you show us the ICLA you have already signed, I will try to merge your first PR |
|
@vongosling |
|
@leonchen83 thanks for your Redis contribution, I have merged. What is the next plan ? Could you introduce some success use case for this project? |
What is the purpose of the change
Develop sub-project rocketmq-redis-replicator
JIRA: https://issues.apache.org/jira/browse/ROCKETMQ-193.
Brief change log
add sub-project rocketmq-redis-replicator
Verifying this change
execute
mvn clean packageto verify.Follow this checklist to help us incorporate your contribution quickly and easily:
mvn -B clean apache-rat:check findbugs:findbugs checkstyle:checkstyleto make sure basic checks pass. Runmvn clean install -DskipITsto make sure unit-test pass. Runmvn clean test-compile failsafe:integration-testto make sure integration-test pass.