The swiss army knife of live data migrations
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.travisci Run update before installing python3-pip Nov 12, 2018
copydb Added an info line to clarify that copydb is done Aug 27, 2018
docs Fixed tutorial to be more copy pasteable Aug 13, 2018
examples Minor fixes Aug 2, 2018
sharding First implementing the capability to resume Oct 2, 2018
test
testhelpers
tlaplus Updated Ghostferry TLA model with BinlogWriter Sep 18, 2018
vendor
webui Changed verifier interface Feb 13, 2018
.gitignore Setup dev/CI environment Jul 27, 2017
.travis.yml
Gopkg.lock Handle SIGTERM/SIGINT and dump state to stdout Sep 25, 2018
Gopkg.toml
LICENSE
Makefile
README.md Added percona live talk + presenter notes Aug 8, 2018
batch_writer.go Refactored Ferry to create components Oct 2, 2018
binlog_streamer.go Record binlog position on connect Nov 27, 2018
binlog_writer.go
compression_verifier.go use compression verifier in sharding.go Aug 10, 2018
config.go Remove AllowResumeWithMismatchedGhostferryVersion Oct 30, 2018
control_server.go
cursor.go Minor fixes Oct 15, 2018
data_iterator.go Fixes from feedback Nov 20, 2018
dev.yml
dml_events.go Pass the current BinlogPosition to DMLEvent Sep 18, 2018
docker-compose.yml Updated example configuration to 3 nodes Jun 5, 2018
error_handler.go Minor fixes Sep 25, 2018
ferry.go
filter.go Added column argument to BuildSelect Feb 1, 2018
http_callback.go
iterative_verifier.go
metrics.go Flush reloc metrics before exiting Mar 15, 2018
row_batch.go Changed TableCopy -> DMLEventBase Feb 27, 2018
state_tracker.go Fixes from feedback Nov 20, 2018
status.go
table_schema_cache.go Fix maxPk vs. tableIsEmpty race Jan 25, 2018
throttler.go Handle empty throttler results Jun 12, 2018
utils.go WithRetriesContext changed context check location Aug 27, 2018
verifier.go copy primary key tables at the end of cutover Apr 25, 2018
wait_until_replica_is_caught_up_to_master.go Refactored replication config checking into Ferry Jun 5, 2018

README.md

Ghostferry

Ghostferry is a library that enables you to selectively copy data from one mysql instance to another with minimal amount of downtime.

It is inspired by Github's gh-ost, although instead of copying data from and to the same database, Ghostferry copies data from one database to another and have the ability to only partially copy data.

There is an example application called ghostferry-copydb included (under the copydb directory) that demonstrates this library by copying an entire database from one machine to another.

Talk to us on IRC at irc.freenode.net #ghostferry.

Overview of How it Works

An overview of Ghostferry's high-level design is expressed in the TLA+ specification, under the tlaplus directory. It maybe good to consult with that as it has a concise definition. However, the specification might not be entirely correct as proofs remain elusive.

On a high-level, Ghostferry is broken into several components, enabling it to copy data. This is documented at https://shopify.github.io/ghostferry/master/technicaloverview.html

Development Setup

Install:

  • Have Docker installed
  • Clone the repo
  • docker-compose up -d mysql-1 mysql-2

Run tests:

  • make test

Test copydb:

  • make copydb && ghostferry-copydb -verbose examples/copydb/conf.json
  • For a more detailed tutorial, see the documentation.