Skip to content
Sync data between persistence engines, like ETL only not stodgy
Branch: master
Clone or download
codepope Fixes for getting latest Go and more
Order fixed to cd into transporter before building and get dependencies.
Latest commit 9e154e7 May 14, 2018
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.github
adaptor fix go lints, oklog/oklog -> oklog/run, add Dockerfile (#459) Mar 13, 2018
client
cmd/transporter fix go lints, oklog/oklog -> oklog/run, add Dockerfile (#459) Mar 13, 2018
commitlog
config
events
function run golint on each package being tested with set_exit_status flag (#410) Aug 30, 2017
integration_tests new write_timeout option available in pipeline configuration (#383) Jun 24, 2017
log Migrate glide => dep (#458) Mar 13, 2018
message
offset run golint on each package being tested with set_exit_status flag (#410) Aug 30, 2017
pipe run golint on each package being tested with set_exit_status flag (#410) Aug 30, 2017
pipeline Log sink pipe errors from write (#427) Dec 12, 2017
scripts sudo pip install for now (#414) Sep 11, 2017
vendor fix go lints, oklog/oklog -> oklog/run, add Dockerfile (#459) Mar 13, 2018
.ansible-provision.yml
.ansible-requirements.yml
.gitattributes commit log compaction (#367) May 12, 2017
.gitignore add type check for map slice and mejson.M/S (#315) Mar 21, 2017
.travis.yml
AUTHORS
CHANGELOG.md
CONTRIBUTING.md added open source material for contributing Dec 12, 2014
DESIGN.md official DESIGN doc covering the basics as well as the commit log (#323) Jun 23, 2017
Dockerfile fix go lints, oklog/oklog -> oklog/run, add Dockerfile (#459) Mar 13, 2018
Gopkg.lock fix go lints, oklog/oklog -> oklog/run, add Dockerfile (#459) Mar 13, 2018
Gopkg.toml
LICENSE First public release Dec 12, 2014
MAINTAINERS update maintainers Jan 27, 2017
README.md
READMEMACOS.md
READMEVAGRANT.md
READMEWINDOWS.md Fixes for getting latest Go and more May 14, 2018
Vagrantfile
release.fish add windows to release script Feb 1, 2017

README.md

Build Status Build Status Go Report Card codecov Docker Repository on Quay Gitter

Compose Transporter helps with database transformations from one store to another. It can also sync from one to another or several stores.

Transporter

About

Transporter allows the user to configure a number of data adaptors as sources or sinks. These can be databases, files or other resources. Data is read from the sources, converted into a message format, and then send down to the sink where the message is converted into a writable format for its destination. The user can also create data transformations in JavaScript which can sit between the source and sink and manipulate or filter the message flow.

Adaptors may be able to track changes as they happen in source data. This "tail" capability allows a Transporter to stay running and keep the sinks in sync.

BETA Feature

As of release v0.4.0, transporter contains support for being able to resume operations after being stopped. The feature is disabled by default and can be enabled with the following:

source = mongodb({"uri": "mongo://localhost:27017/source_db"})
sink = mongodb({"uri": "mongo://localhost:27017/sink_db"})
t.Config({"xlog_dir":"/data/transporter"})
  .Source("source", source)
  .Save("sink", sink)

When using the above pipeline, all messages will be appended to a commit log and successful processing of a message is handled via consumer/sink offset tracking.

Below is a list of each adaptor and its support of the feature:

+---------------+-------------+----------------+
|    adaptor    | read resume | write tracking |
+---------------+-------------+----------------+
| elasticsearch |     N/A     |       X        | 
|     file      |             |       X        | 
|    mongodb    |      X      |       X        | 
|  postgresql   |             |       X        | 
|   rabbitmq    |      X      |                | 
|   rethinkdb   |             |       X        | 
+---------------+-------------+----------------+

Downloading Transporter

The latest binary releases are available from the Github Repository

Adaptors

Each adaptor has its own README page with details on configuration and capabilities.

Native Functions

Each native function can be used as part of a Transform step in the pipeline.

Commands

init

transporter init [source adaptor name] [sink adaptor name]

Generates a basic pipeline.js file in the current directory.

Example

$ transporter init mongodb elasticsearch
$ cat pipeline.js
var source = mongodb({
  "uri": "${MONGODB_URI}"
  // "timeout": "30s",
  // "tail": false,
  // "ssl": false,
  // "cacerts": ["/path/to/cert.pem"],
  // "wc": 1,
  // "fsync": false,
  // "bulk": false,
  // "collection_filters": "{}",
  // "read_preference": "Primary"
})

var sink = elasticsearch({
  "uri": "${ELASTICSEARCH_URI}"
  // "timeout": "10s", // defaults to 30s
  // "aws_access_key": "ABCDEF", // used for signing requests to AWS Elasticsearch service
  // "aws_access_secret": "ABCDEF" // used for signing requests to AWS Elasticsearch service
})

t.Source(source).Save(sink)
// t.Source("source", source).Save("sink", sink)
// t.Source("source", source, "namespace").Save("sink", sink, "namespace")
$

Edit the pipeline.js file to configure the source and sink nodes and also to set the namespace.

about

transporter about

Lists all the adaptors currently available.

Example

elasticsearch - an elasticsearch sink adaptor
file - an adaptor that reads / writes files
mongodb - a mongodb adaptor that functions as both a source and a sink
postgres - a postgres adaptor that functions as both a source and a sink
rabbitmq - an adaptor that handles publish/subscribe messaging with RabbitMQ 
rethinkdb - a rethinkdb adaptor that functions as both a source and a sink

Giving the name of an adaptor produces more detail, such as the sample configuration.

Example

transporter about postgres
postgres - a postgres adaptor that functions as both a source and a sink

 Sample configuration:
{
  "uri": "${POSTGRESQL_URI}"
  // "debug": false,
  // "tail": false,
  // "replication_slot": "slot"
}

run

transporter run [-log.level "info"] <application.js>

Runs the pipeline script file which has its name given as the final parameter.

test

transporter test [-log.level "info"] <application.js>

Evaluates and connects the pipeline, sources and sinks. Establishes connections but does not run. Prints out the state of connections at the end. Useful for debugging new configurations.

xlog

The xlog command is useful for inspecting the current state of the commit log. It contains 3 subcommands, current, oldest, and offset, as well as a required flag -xlog_dir which should be the path to where the commit log is stored.

NOTE the command should only be run against the commit log when transporter is not actively running.

transporter xlog -xlog_dir=/path/to/dir current
12345

Returns the most recent offset appended to the commit log.

transporter xlog -xlog_dir=/path/to/dir oldest
0

Returns the oldest offset in the commit log.

transporter xlog -xlog_dir=/path/to/dir show 0
offset    : 0
timestamp : 2017-05-16 11:00:20 -0400 EDT
mode      : COPY
op        : INSERT
key       : MyCollection
value     : {"_id":{"$oid":"58efd14b60d271d7457b4f24"},"i":0}

Prints out the entry stored at the provided offset.

offset

The offset command provides access to current state of each consumer (i.e. sink) offset. It contains 4 subcommands, list, show, mark, and delete, as well as a required flag -xlog_dir which should be the path to where the commit log is stored.

transporter offset -xlog_dir=/path/to/dir list
+------+---------+
| SINK | OFFSET  |
+------+---------+
| sink | 1103003 |
+------+---------+

Lists all consumers and their associated offset in xlog_dir.

transporter offset -xlog_dir=/path/to/dir show sink
+-------------------+---------+
|     NAMESPACE     | OFFSET  |
+-------------------+---------+
| newCollection     | 1102756 |
| testC             | 1103003 |
| MyCollection      |  999429 |
| anotherCollection | 1002997 |
+-------------------+---------+

Prints out each namespace and its associated offset.

transporter offset -xlog_dir=/path/to/dir mark sink 1
OK

Rewrites the namespace offset map based on the provided offset.

transporter offset -xlog_dir=/path/to/dir delete sink
OK

Removes the consumer (i.e. sink) log directory.

flags

-log.level "info" - sets the logging level. This is application logging and is unrelated to the commit log. Default is info; can be debug or error.

Building Transporter

Essentials

go build ./cmd/transporter/...

Building guides

macOS Windows Vagrant

Transporter Resources

Contributing to Transporter

Want to help out with Transporter? Great! There are instructions to get you started here.

Licensing

Transporter is licensed under the New BSD License. See LICENSE for full license text.

Support and Guarantees

Compose does not provide support nor guarantee stability or functionality of this tool. Please take adequate caution when using Transporter to ensure that it's the right tool for the job. Transporter may not account for failure scenarios that could lead to unexpected behavior. Always take backups, always test in dev, and always feel free to submit a PR with enhancements, features, and bug fixes.

You can’t perform that action at this time.