Maxwell's daemon, a mysql-to-json kafka producer
Switch branches/tags
Clone or download
Pull request Compare This branch is 4 commits ahead, 390 commits behind zendesk:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
bin
build
docs
img
src
.dockerignore
.editorconfig
.gitattributes
.gitignore
.travis.yml
CHANGELOG.md
Dockerfile
LICENSE
Makefile
README.md
config.properties.example
kinesis-producer-library.properties.example
pom.xml

README.md

This is Maxwell's daemon, an application that reads MySQL binlogs and writes row updates to Kafka, Kinesis, RabbitMQ, Google Cloud Pub/Sub, or Redis as JSON. Maxwell has a low operational bar and produces a consistent, easy to ingest stream of updates. It allows you to easily "bolt on" some of the benefits of stream processing systems without going through your entire code base to add (unreliable) instrumentation points. Common use cases include ETL, cache building/expiring, metrics collection, search indexing and inter-service communication.

  • Can do SELECT * from table (bootstrapping) initial loads of a table.
  • supports automatic position recover on master promotion
  • flexible partitioning schemes for Kakfa - by database, table, primary key, or column
  • Maxwell pulls all this off by acting as a full mysql replica, including a SQL parser for create/alter/drop statements (nope, there was no other way).

→ Download: https://github.com/zendesk/maxwell/releases/download/v1.12.0/maxwell-1.12.0.tar.gz
→ Source: https://github.com/zendesk/maxwell


  mysql> insert into `test`.`maxwell` set id = 1, daemon = 'Stanislaw Lem';
  maxwell: {
    "database": "test",
    "table": "maxwell",
    "type": "insert",
    "ts": 1449786310,
    "xid": 940752,
    "commit": true,
    "data": { "id":1, "daemon": "Stanislaw Lem" }
  }
  mysql> update test.maxwell set daemon = 'firebus!  firebus!' where id = 1;
  maxwell: {
    "database": "test",
    "table": "maxwell",
    "type": "update",
    "ts": 1449786341,
    "xid": 940786,
    "commit": true,
    "data": {"id":1, "daemon": "Firebus!  Firebus!"},
    "old":  {"daemon": "Stanislaw Lem"}
  }