A tool that can drive a replication between Sync Gateway instances
Go
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
cli
.drone.sec
.drone.yml
.gitignore
LICENSE-APACHE2.txt
README.md
attachment.go
changes_feed_parameters.go
changes_feed_parameters_test.go
continuous_replication.go
continuous_replication_test.go
logging.go
replication_event.go
replication_notification.go
replication_parameters.go
replication_parameters_test.go
replication_state.go
replication_stats.go
rest_data.go
rest_data_test.go
sequence_number.go
synctube.go
synctube_test.go

README.md

Join the chat at https://gitter.im/couchbase/mobile Build Status GoDoc Coverage Status Go Report Card codebeat badge

sg-replicate is a tool that can drive a replication between Sync Gateway instances.

It was created because the Sync Gateway can only serve as a passive replication target, but it does not offer a mechanism to drive a replication. This tool attempts to fill that gap.

architecture

Sync Gateway integration

sg-replicate has been integrated into Sync Gateway itself, and that is probably how you want to use it, unless you have a specific need to run it standalone. See the documentation on Sync Gateway Replications for usage instructions.

Quickstart - build from source

Note: there is no need to clone this repo, because the go get command below will take care of that for you

  • Install Go and define $GOPATH environment variable (eg, export GOPATH=~/gocode)
  • $ go get -u -v github.com/couchbaselabs/sg-replicate/... -- clones this repo and all source dependencies into $GOPATH/src
  • $ cd ${GOPATH}/src/github.com/couchbaselabs/sg-replicate/cli
  • $ cp config.json.example config.json
  • Customize config.json to contain the server URLs of your source and target server.
  • Build and run:
$ go build && ./cli

Notes on command behavior:

  • If you have any continuous replications, the command will block indefinitely, and only exit if there is a non-recoverable error with a continuous replication.
  • If you only have oneshot replications defined, the command will exit once they have all completed.

Features

  • Json configuration file to specify replications
  • Supports multiple replications running concurrently
  • Can run both OneShot and Continuous replications
  • Does not store anything persistently
  • Stateless -- can be interrupted/restarted anytime without negative side effects.
  • Filter replications using channels.

Documentation

GoDoc

Sample Configuration

{
    "changes_feed_limit": 50,
    "continuous_retry_time_ms": 15000,
    "replications":{
        "db-local":{
            "source_url": "http://localhost:4985",
            "source_db": "db",
            "target_db": "db-copy",
            "lifecycle": "oneshot"
        },
        "grocery-sync-local":{
            "source_url": "http://localhost:4985",
            "target_url": "http://sync.couchbasecloud.com",
            "source_db": "grocery-sync",
            "target_db": "grocery-sync",
            "lifecycle": "continuous",
            "channels": ["lists", "items"]
        }
    }
}

Configuration fields

  • changes_feed_limit -- the number of changes to fetch in each "batch". Setting this to a larger value will increase memory consumption.
  • continuous_retry_time_ms -- the amount of time to wait (in milliseconds) after an aborted replication before retrying. (only applicable to continuous replications)
  • replications -- a "map" of replications, where each replication has a unique name. they will be run in the order given in this file.
  • source_url -- url of source sync gateway, without the db name in the url. Can point to admin port (:4985) or user port (:4984), but be aware if you point it to the user port, you will probably need to set a username/pass in the url and will only replicate docs visible to that user.
  • source_db -- the source db to pull from.
  • target_url -- url of target sync gateway, without the db name in the url. If omitted, it will be assumed that it's the same as the source_url See source_url for discussion of which port to use.
  • target_db -- the target db to push to.
  • channels -- the set of channels that replication should be restricted to.
  • disabled -- is this replication currently disabled? (true | false)
  • lifecycle -- possible values: oneshot or continuous.
    • oneshot replications will be run inline (synchronously), and it will not start the following replications until the oneshot replication completes.
    • continuous replications are started in parallel with other continuous replications, and run indefinitely until they have a non-recoverable error.

Filing Issues

Please file issues in the Sync Gateway issue tracker. sg-replicate is now an officially supported component of Sync Gateway.

Todo

  • Logs are difficult to disentangle when multiple replications are running -- workaround: use separate instances and separate config files for each replication
  • Clean up API to only expose what's necessary
  • Attachments handling should be made to be more efficient. Currently, attachment data is temporarily stored in memory before it is pushed to the target server.
  • targetCheckpointAddress needs to take more things into account when generating checkpoint address.