StorageTapper is a scalable realtime MySQL change data streaming and transformation service.
Service reads data from MySQL, transforms it into an Avro schema serialized format, and publishes these events to Kafka. Consumers can then consume these events directly from Kafka.
- Producing events to Kafka
- Automatic work distribution between instances
- Avro output format
- JSON output format
- HTTP endpoints to control
- Tables to be ingested
- Output schema
- Database address resolver
- Snapshot is taken from slave to reduce load on master
- Binlogs streaming from master for better SLA
- Throttling when taking snapshot
- Tables must have a primary key
Build & Install
Get build dependencies
go get github.com/Masterminds/glide go get github.com/alecthomas/gometalinter
It's recommended to have local MySQL and Kafka installed, so as many tests depend on them. Tweak development config to correspond your setup. WARNING: Tests run dangerous commands like RESET MASTER and DROP DATABASE, so for tests don't use any MySQL instances with precious data.
Debian & Ubuntu
cd storagetapper make deb && dpkg -i ../storagetapper_1.0_amd64.deb
cd storagetapper make test && make install
Storagetapper loads configuration from the following files and location in the given order:
/etc/storagetapper/base.yaml /etc/storagetapper/production.yaml $(HOME)/base.yaml $(HOME)/production.yaml $(STORAGETAPPER_CONFIG_DIR)/base.yaml $(STORAGETAPPER_CONFIG_DIR)/production.yaml
Available options described in Options section
This software is licensed under the MIT License.