Skip to content
MySQLStreamer is a database change data capture and publish system.
Branch: master
Clone or download
Latest commit 568b807 Jan 11, 2017
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
acceptance added apache LICENSE v2 to replication_handler Oct 31, 2016
bin bumped up the version of venv-update May 9, 2016
devdbs added apache LICENSE v2 to replication_handler Oct 31, 2016
docs added apache LICENSE v2 to replication_handler Oct 31, 2016
logs Make replication handler tests pass to get the build building again Mar 2, 2016
replication_handler remove references of force_avoid_internal_packages in replication_han… Dec 30, 2016
replication_handler_testing remove references of force_avoid_internal_packages in replication_han… Dec 30, 2016
requirements.d Rolled back reqs change Nov 18, 2016
schema
tests remove references of force_avoid_internal_packages in replication_han… Dec 30, 2016
.coveragerc Now create mysql dump before and after schema_events for backup+futur… Nov 30, 2016
.coveragerc-opensource fixed an issue with parse replication stream test Oct 20, 2016
.dockerignore Enables replication handler to connect to alternative refresh dbs Nov 21, 2016
.gitignore [DATAPIPE-1456] Fix for alter table statement setting row_format Aug 29, 2016
.pre-commit-config.yaml added apache LICENSE v2 to replication_handler Oct 31, 2016
.rat-excludes added apache LICENSE v2 to replication_handler Oct 31, 2016
Dockerfile setting up replication handler files to support open source dev Oct 19, 2016
Dockerfile-opensource setting up replication handler files to support open source dev Oct 19, 2016
LICENSE added apache LICENSE v2 to replication_handler Oct 31, 2016
Makefile Rolled back reqs change Nov 18, 2016
Makefile-opensource attach RH to tmux pane 1.0 Oct 28, 2016
MySQLStreamerWorking.gif Testing gif Nov 17, 2016
NOTICE added apache LICENSE v2 to replication_handler Oct 31, 2016
README.md Testing gif Nov 17, 2016
config-env-changelog-itest.yaml setting kafka_cluster_type in changelog itest to datapipe Sep 23, 2016
config-env-dev.yaml Add yelp_heartbeat to schema_blacklist configs Dec 12, 2016
config-env-itest-opensource.yaml Add yelp_heartbeat to schema_blacklist configs Dec 12, 2016
config-env-itest.yaml Add yelp_heartbeat to schema_blacklist configs Dec 12, 2016
config.yaml Add yelp_heartbeat to schema_blacklist configs Dec 12, 2016
connection_sets.yaml more fix Jul 23, 2015
docker-compose-opensource.yml remove references of force_avoid_internal_packages in replication_han… Dec 30, 2016
docker-compose.yml create open source version of parse_replication_stream in RH Oct 7, 2016
fig.yml small fix Jul 16, 2015
interactive_streamer.py added apache LICENSE v2 to replication_handler Oct 31, 2016
requirements-opensource.txt added apache LICENSE v2 to replication_handler Oct 31, 2016
requirements.txt Start using public pymysqlreplication Nov 30, 2016
setup.py Merge branch 'master' into DATAPIPE-2008-abrar-license Nov 4, 2016
tox-opensource.ini added apache LICENSE v2 to replication_handler Oct 31, 2016
tox.ini Rolled back reqs change Nov 18, 2016
yelp_conn_generic.yaml add yelp_conn Mar 12, 2015

README.md

MySQL Streamer

What is it?

MySQLStreamer is a database change data capture and publish system. It’s responsible for capturing each individual database change, enveloping them into messages and publishing to Kafka.

Read More

How to download

git clone git@github.com:Yelp/mysql_streamer.git

Requirements

For Ubuntu 14.04

run apt-get update && apt-get upgrade -y && apt-get install -y \
   build-essential \
   python-dev \
   libmysqlclient-dev \
   python-pkg-resources \
   python-setuptools \
   python-virtualenv \
   python-pip \
   libpq5 \
   libpq-dev \
   wget \
   language-pack-en-base \
   uuid-dev \
   git-core \
   mysql-client-5.5

Tests

Running unit tests

make -f Makefile-opensource test

Running integration tests

make -f Makefile-opensource itest

Demo

make -f Makefile-opensource interactive-streamer

MySQL Streamer Demo

In the bottom pane, you can execute any mysql statement on a barebones structure we’ve set up. Try creating a table and inserting into it to see it be sent (seen in the replication handler logs in the top right), and then received messages in the top left (seen in the kafka tailer’s logs). The received messages will probably look a little funky - this is because they’re avro-encoded.

Disclaimer

We're still in the process of setting up this service as a stand-alone. There may be additional work required to run MySQL Streamer instances and integrate with other applications.

License

MySQL Streamer is licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0

Contributing

Everyone is encouraged to contribute to MySQL Streamer by forking the Github repository and making a pull request or opening an issue.

You can’t perform that action at this time.