Permalink
Browse files

Initial commit

  • Loading branch information...
0 parents commit 3f3fbb7900c4a36b72c67792e50892d1fad9fdb2 @y310 committed Nov 28, 2012
Showing with 572 additions and 0 deletions.
  1. +17 −0 .gitignore
  2. +4 −0 Gemfile
  3. +5 −0 Guardfile
  4. +22 −0 LICENSE.txt
  5. +154 −0 README.md
  6. +1 −0 Rakefile
  7. +28 −0 kodama.gemspec
  8. +1 −0 lib/kodama.rb
  9. +207 −0 lib/kodama/client.rb
  10. +3 −0 lib/kodama/version.rb
  11. +129 −0 spec/lib/client_spec.rb
  12. +1 −0 spec/spec_helper.rb
@@ -0,0 +1,17 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp
@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in kodama.gemspec
+gemspec
@@ -0,0 +1,5 @@
+guard 'rspec', :version => 2, :cli => '-c' do
+ watch(%r{^spec/.+_spec\.rb$})
+ watch(%r{^lib/(.+)\.rb$}) { |m| "spec/lib/#{m[1]}_spec.rb" }
+ watch('spec/spec_helper.rb') { "spec" }
+end
@@ -0,0 +1,22 @@
+Copyright (c) 2012 Yusuke Mito
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
154 README.md
@@ -0,0 +1,154 @@
+# Kodama
+
+Kodama is a MySQL replication listener based on [ruby-binlog](https://bitbucket.org/winebarrel/ruby-binlog/overview).
+Kodama provides a simple DSL to easily write your own replication listener.
+
+## Features
+
+- Provides simple DSL for writing binlog event handlers
+- Automatically restarts from the saved binlog position
+- Attempts to reconnect to MySQL when the connection is somehow teminated
+
+These features allow developers to focus on writing their own replication logic rather than having to spend time figuring things out.
+
+## Kodama Benefits
+
+Kodama can be used to replicate MySQL updates to other data stores, arbitrary software or even a flat file. The sole purpose of Kodama is to provide a convenient way to reflect the database updates to other components in your system.
+
+- Replicate from MySQL to Postgres
+- Replicate between tables with different schema
+- Sync production data to development DB while masking privacy information
+- Realtime full text index update
+
+## Dependencies
+
+This gem links against MySQL's libreplication C shared library. You need to first install the [mysql-replication-listener](https://launchpad.net/mysql-replication-listener) package.
+
+But official repository has some bugs. It is recommended to use [winebarrel's patched version](https://bitbucket.org/winebarrel/ruby-binlog/downloads) (There are rpm package and homebrew formula).
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+ gem 'kodama'
+
+And then execute:
+
+ $ bundle
+
+Or install it yourself as:
+
+ $ gem install kodama
+
+### binlog_format
+
+It is recommended to set the mysqld binlog_format option to ``ROW``. This is because the ``ROW`` format allows Kodama to pickup every single updates made to the database.
+
+```sql
+SET GLOBAL binlog_format = 'ROW';
+```
+
+## Usage
+
+### Simple client
+
+```ruby
+require 'kodama'
+
+Kodama::Client.start(:host => '127.0.0.1', :username => 'user') do |c|
+ c.binlog_position_file = 'position.log'
+ c.log_level = :info # [:debug|:info|:warn|:error|:fatal]
+ c.connection_retry_limit = 100 # times
+ c.connection_retry_wait = 3 # second
+
+ c.on_query_event do |event|
+ p event.query
+ end
+
+ c.on_row_event do |event|
+ p event.rows
+ end
+end
+```
+
+### Replicate to redis
+
+```ruby
+require 'rubygems'
+require 'kodama'
+require 'thread'
+require 'json'
+require 'redis'
+
+class Worker
+ attr_accessor :queue
+ def initialize
+ @queue = Queue.new
+ @redis = Redis.new
+ end
+
+ def start
+ Thread.start do
+ loop do
+ event = @queue.pop
+ record_id = get_row(event)[0] # first column is id
+ @redis.set "#{event.table_name}_#{record_id}", event.rows.to_json
+ end
+ end
+ end
+
+ def get_row(event)
+ case event.event_type
+ when /Write/, /Delete/
+ event.rows[0] # [row]
+ when /Update/
+ event.rows[0][1] # [[old_row, new_row]]
+ end
+ end
+end
+
+
+worker = Worker.new
+worker.start
+
+Kodama::Client.start(:host => '127.0.0.1', :username => 'user') do |c|
+ c.binlog_position_file = 'position.log'
+
+ c.on_row_event do |event|
+ worker.queue << event
+ end
+end
+```
+
+## Configuration
+
+### binlog_position_file
+
+Sets the filename to save the binlog position.
+Kodama will read this file and resume listening from the stored position.
+
+### log_level
+
+Set logger's log level.
+It accepts ``:debug``, ``:info``, ``:warn``, ``:error``, ``:fatal``.
+
+### connection_retry_limit, connection_retry_wait
+
+If for some reason the connection to MySQL is terminated, Kodama will attempt to reconnect ``connection_retry_limit`` times, while waiting ``connection_retry_wait`` seconds between attempts.
+
+## Authors
+
+Yusuke Mito, Genki Sugawara
+
+## Based On
+
+- [ruby-binlog](https://bitbucket.org/winebarrel/ruby-binlog)
+- [mysql-replication-listener](https://launchpad.net/mysql-replication-listener)
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request
@@ -0,0 +1 @@
+require "bundler/gem_tasks"
@@ -0,0 +1,28 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'kodama/version'
+
+Gem::Specification.new do |gem|
+ gem.name = "kodama"
+ gem.version = Kodama::VERSION
+ gem.authors = ["Yusuke Mito"]
+ gem.email = ["y310.1984@gmail.com"]
+ gem.description = %q{ruby-binlog based MySQL replication listener}
+ gem.summary = %q{ruby-binlog based MySQL replication listener}
+ gem.homepage = "https://github.com/y310/kodama"
+
+ gem.files = `git ls-files`.split($/)
+ gem.executables = gem.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
+ gem.test_files = gem.files.grep(%r{^(test|spec|features)/})
+ gem.require_paths = ["lib"]
+
+ gem.add_dependency 'ruby-binlog', '>= 0.1.8'
+
+ gem.add_development_dependency 'rake'
+ gem.add_development_dependency 'rspec'
+ gem.add_development_dependency 'pry'
+ gem.add_development_dependency 'pry-nav'
+ gem.add_development_dependency 'guard-rspec', '2.1.2'
+ gem.add_development_dependency 'rb-fsevent', '~> 0.9.1'
+end
@@ -0,0 +1 @@
+require 'kodama/client'
Oops, something went wrong.

0 comments on commit 3f3fbb7

Please sign in to comment.