Skip to content
MongoShake is a universal data replication platform based on MongoDB's oplog. Redundant replication and active-active replication are two most important functions. 基于mongodb oplog的集群复制工具,可以满足迁移和同步的需求,进一步实现灾备和多活功能。
Branch: develop
Clone or download
Latest commit 2d57b89 May 17, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
bin remove receiver Mar 19, 2019
conf polish conf comment Apr 12, 2019
resources Merge pull request #108 from aliyun/feature-1.4 Feb 22, 2019
scripts
src Merge pull request #132 from alibaba/feature-1.4 Apr 16, 2019
tools first commit Jun 27, 2018
.gitignore add binary in the bin directory Mar 18, 2019
ChangeLog v1.4.6: fix bug of batched oplog size exceeding 16MB Apr 12, 2019
LICENSE first commit Jun 27, 2018
Makefile first commit Jun 27, 2018
README.md Update README.md May 17, 2019
build.sh

README.md

This is a brief introduction of Mongo-Shake, please visit english wiki or chinese wiki if you want to see more details including architecture, data flow, performance test, business showcase and so on.

Mongo-Shake


Mongo-Shake is developed and maintained by Nosql Team in Alibaba-Cloud.
Mongo-Shake is a universal platform for services based on MongoDB's oplog. It fetches oplog from source mongo database, and replays in the target mongo database or sends to other ends in different tunnels. If the target side is mongo database which means replay oplog directly, it's like a syncing tool that used to copy data from source MongoDB to another MongoDB to build redundant replication or active-active replication. Except for this direct way, there are others tunnel types such like rpc, file, tcp, kafka. Receivers wrote by users must define their own interfaces to connecting to these tunnels respectively. Users can also define there own tunnel type which is pluggable. If connecting to a third-party message middleware like kafka, the consumer can get the subscriber data in an asynchronous way in pub/sub module flexibly. Here comes general data flow,
pic1
The source can be either single mongod, replica set or sharding while target can be mongod or mongos. If the source is replica set, we suggest fetching data from secondary/hidden to ease the primary pressure. If the source is sharding, every shard should connect to Mongo-Shake. There can be several mongos on the target side to keep high availability, and different data will be hashed and written to different mongos.

Parallel Replication


There are three options for parallel replication which we call 'shad_key': id, collection and auto. id means the concurrency granularity is document while collection means the granularity is collection/table. auto option is decided by if there has unique index of any collection, it will change to collection with unique index exist otherwise id.

High Availability


Mongo-Shake periodically persistent its context into register center which by default is the source database. Currently, the context is checkpoint which marks the position of successfully replay oplog.
Hypervisor mechanism is also supported so that it will restart immediately when dies(master_quorum in configuration).

Filter


Support filtering database and collection namespace with whitelist and blacklist.

Global ID


In Aliyun internal version, global id(also called gid) is supported which marks the id of the database. It can be used to avoid loop when two databases become backup of each other. Mongo-Shake only fetches the oplogs equal to source database id, all the oplogs are be fetched when no gid gave. For current opensource version, it's not supported limited by the modification of MongoDB kernel. If you want to build active-active replication without gid supported, please visit FAQ document to see more details.

Tunnel


As mentioned above, we support several tunnel types such like: rcp, tcp, file, kafka, mock and direct. rpc and tcp means connecting to receiver synchronously by net/rcp and TCP respectively; file means writing output into file; kafka is an asynchronous way of sending the message; mock is used by testing that throws away all the data; direct means writing into target MongoDB directly. Users can also add or modify current tunnel type.
We offer receiver to connect to different tunnels like: rpc, tcp, file, mock and kafka. Please visit FAQ document to see more details.

Compressor


Gzip, zlib, deflate compressor are supported in batched oplogs before sending.

Monitor & Debug


User can monitor or debug Mongo-Shake through RESTful API, please visit FAQ document to see more details.

Other Details


Mongo-Shake uses mgo.v2 library which is a modification version of mgo.v2 to fetch oplogs from source MongoDB which is later than the given timestamp in configuration. Then, it filters oplogs based on whitelist, blacklist, and gid. All the oplogs will be transferred at least once which is acceptable because of idempotent of oplog DML. We use seq and ack to make sure the package is received which is similar to the sequence and acknowledgment numbers in TCP.
The oplogs are batched together in the handling pipeline.
Users can adjust the worker concurrency and executor concurrency according to the different environment.
Please see the detail documents listed at the beginning if you want to see more details.

Code branch rules

version rules: a.b.c.

  • a: major version
  • b: minor version. even number means stable version. e.g. 1.2.x, 1.4.x, 2.0.x are stable while 1.5.x, 2.1.x aren't.
  • c: bugfix version
branch name rules
master master branch, do not allowed push code. store the latest stable version.
develop(main branch) develop branch. all the bellowing branches fork from this.
feature-* new feature branch. forked from develop branch and then merge back after finish developing, testing, and code review.
bugfix-* bugfix branch. forked from develop branch and then merge back after finish developing, testing, and code review.
improve-* improvement branch. forked from develop branch and then merge back after finish developing, testing, and code review.

tag rules: add tag when releasing: "release-v{version}-{date}". for example: "release-v1.0.2-20180628"

Usage


  • git clone https://github.com/aliyun/mongo-shake.git
  • cd mongo-shake/src/vendor
  • GOPATH=`pwd`/../..; govendor sync #please note: must install govendor first and then pull all dependencies
  • cd ../../ && ./build.sh
  • ./bin/collector -conf=conf/collector.conf #please note: user must modify collector.conf first to match needs. You can also use "start.sh" script which supports hypervisor mechanism in Linux OS only.

Shake series tool


We also provide some tools for synchronization in Shake series.

Plus, we have a WeChat group so that users can join and discuss, but the group user number is limited. So please add my WeChat number: vinllen_xingge first, and I will add you to this group.

You can’t perform that action at this time.