Official mydumper project
Switch branches/tags
Clone or download
Latest commit a1ddcba May 21, 2018
Permalink
Failed to load latest commit information.
cmake/modules add so.1 suffix for xmake to find libatomic on Debian (and Ubuntu?) (#35 Aug 2, 2017
docs fixes #123 - regex examples Apr 27, 2018
.bzrignore Added .partial to metadata, removed when finished Jun 25, 2013
CMakeLists.txt set version 0.9.5 May 4, 2018
LICENSE Create LICENSE (#57) Sep 8, 2017
README.md README update May 4, 2018
binlog.c * Don't fail on deprecated glib functions, I like this to build on ol… Oct 20, 2012
binlog.h Fix thread shutdown issues Jun 13, 2011
common.h Fix wrong WITH_SSL naming May 21, 2018
config.h.in Fix wrong WITH_SSL naming May 21, 2018
connection.c fix mydumper main connection SSL May 3, 2018
connection.h Support SSL MySQL connections (#36) Apr 26, 2018
g_unix_signal.c Add basic signal handler code (a modified version of the code used in… Jun 2, 2011
g_unix_signal.h Add basic signal handler code (a modified version of the code used in… Jun 2, 2011
getPassword.c Adding an option in order to not disclose the password on the command… Apr 26, 2018
getPassword.h Adding an option in order to not disclose the password on the command… Apr 26, 2018
mydumper.c fix mydumper main connection SSL May 3, 2018
mydumper.h events, sp and functions support added to mydumper/myloader Nov 28, 2014
myloader.c Support SSL MySQL connections (#36) Apr 26, 2018
myloader.h add myloader binary to .bzrignore May 16, 2011
server_detect.c bug fixes adding support to galera cluster and others Jan 17, 2014
server_detect.h bug fixes adding support to galera cluster and others Jan 17, 2014

README.md

What is mydumper? Why?

  • Parallelism (hence, speed) and performance (avoids expensive character set conversion routines, efficient code overall)
  • Easier to manage output (separate files for tables, dump metadata, etc, easy to view/parse data)
  • Consistency - maintains snapshot across all threads, provides accurate master and slave log positions, etc
  • Manageability - supports PCRE for specifying database and tables inclusions and exclusions

How to install mydumper/myloader?

First get the correct url from the releases section then:

RedHat / Centos

yum install https://github.com/maxbube/mydumper/releases/download/v0.9.5/mydumper-0.9.5-1.el7.x86_64.rpm

Ubuntu / Debian

wget https://github.com/maxbube/mydumper/releases/download/v0.9.5/mydumper_0.9.5-1.xenial_amd64.deb
dpkg -i mydumper_0.9.5-1.xenial_amd64.deb

How to build it?

Run:

cmake .
make

One needs to install development versions of required libaries (MySQL, GLib, ZLib, PCRE): NOTE: you must use the correspondent mysql devel package.

  • Ubuntu or Debian: apt-get install libglib2.0-dev libmysqlclient15-dev zlib1g-dev libpcre3-dev libssl-dev
  • Fedora, RedHat and CentOS: yum install glib2-devel mysql-devel zlib-devel pcre-devel openssl-devel
  • openSUSE: zypper install glib2-devel libmysqlclient-devel pcre-devel zlib-devel
  • MacOSX: port install glib2 mysql5 pcre pkgconfig cmake (You may want to run 'port select mysql mysql5' afterwards)

One has to make sure, that pkg-config, mysql_config, pcre-config are all in $PATH

Binlog dump is disabled by default to compile with it you need to add -DWITH_BINLOG=ON to cmake options

To build against mysql libs < 5.7 you need to disable SSL adding -DWITH_SSL=OFF

How does consistent snapshot work?

This is all done following best MySQL practices and traditions:

  • As a precaution, slow running queries on the server either abort the dump, or get killed
  • Global write lock is acquired ("FLUSH TABLES WITH READ LOCK")
  • Various metadata is read ("SHOW SLAVE STATUS","SHOW MASTER STATUS")
  • Other threads connect and establish snapshots ("START TRANSACTION WITH CONSISTENT SNAPSHOT") ** On pre-4.1.8 it creates dummy InnoDB table, and reads from it.
  • Once all worker threads announce the snapshot establishment, master executes "UNLOCK TABLES" and starts queueing jobs.

This for now does not provide consistent snapshots for non-transactional engines - support for that is expected in 0.2 :)

How to exclude (or include) databases?

Once can use --regex functionality, for example not to dump mysql and test databases:

 mydumper --regex '^(?!(mysql\.|test\.))'

To dump only mysql and test databases:

 mydumper --regex '^(mysql\.|test\.)'

To not dump all databases starting with test:

 mydumper --regex '^(?!(test))'

Of course, regex functionality can be used to describe pretty much any list of tables.