Skip to content
StorageTapper is a scalable realtime MySQL change data streaming and transformation service
Branch: master
Clone or download
efirs Skip flaky throttle test
Signed-off-by: Evgeniy Firsov <firsov@uber.com>
Latest commit ab40c3c Jan 7, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
changelog Reuse lock connection, implement TryLock Sep 21, 2018
config Remove default output format Dec 17, 2017
db
debian Add missing systemd service file and create storagetapper user Oct 1, 2017
doc
encoder
lock Reuse lock connection, implement TryLock Sep 21, 2018
log
metrics Initial commit May 9, 2017
pipe Partially fix staticcheck lint Dec 21, 2017
pool
schema Increase schema package coverage Dec 15, 2017
scripts Update Kafka to 0.11.0.3 Jan 6, 2019
server
shutdown Producer/Consumer reuse parent pipe variables Dec 15, 2017
snapshot
state
streamer
test
throttle
types
util
.gitignore
.travis.yml
LICENSE Initial commit May 9, 2017
Makefile
README.md
codecov.yaml
glide.lock Update msgp library Jan 6, 2019
glide.yaml
main.go
main_test.go
pprof_test.go Fix race condition in tests Oct 27, 2017

README.md

StorageTapper

StorageTapper is a scalable realtime MySQL change data streaming and transformation service.

Service reads data from MySQL, transforms it into an Avro schema serialized format, and publishes these events to Kafka. Consumers can then consume these events directly from Kafka.

Build Status Go Report Card codecov

Features

  • Producing events to Kafka
  • Automatic work distribution between instances
  • Avro output format
  • JSON output format
  • HTTP endpoints to control
    • Tables to be ingested
    • Output schema
    • Database address resolver
  • Snapshot is taken from slave to reduce load on master
  • Binlogs streaming from master for better SLA
  • Throttling when taking snapshot

Limitations

  • Tables must have a primary key

Build & Install

Get build dependencies

go get github.com/Masterminds/glide
go get github.com/alecthomas/gometalinter

It's recommended to have local MySQL and Kafka installed, so as many tests depend on them. Tweak development config to correspond your setup. WARNING: Tests run dangerous commands like RESET MASTER and DROP DATABASE, so for tests don't use any MySQL instances with precious data.

Debian & Ubuntu

cd storagetapper
make deb && dpkg -i ../storagetapper_1.0_amd64.deb

Others

cd storagetapper
make test && make install

Configuration

Storagetapper loads configuration from the following files and location in the given order:

    /etc/storagetapper/base.yaml
    /etc/storagetapper/production.yaml
    $(HOME)/base.yaml
    $(HOME)/production.yaml
    $(STORAGETAPPER_CONFIG_DIR)/base.yaml
    $(STORAGETAPPER_CONFIG_DIR)/production.yaml

Available options described in Options section

License

This software is licensed under the MIT License.

You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.