Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
branch: master
Fetching contributors…

Cannot retrieve contributors at this time

69 lines (46 sloc) 5.402 kB

Jetpants Configuration

Jetpants supports a global configuration file at /etc/jetpants.yaml, as well as per-user configuration files at ~/.jetpants.yaml.

At least one of these files must exist for Jetpants to function properly.

If both exist, the user configuration will be merged on top of the global configuration, this is a “deep” merge. So if the user a “plugins” section this will be combined with the global “plugins” section.

For an example global configuration file, please see the included etc/jetpants.yaml.sample file.

Configuration settings

max_concurrency

Maximum threads to use in import/export operations, or equivalently the maximum connection pool size per database host. You will need to tune this to your specific database hardware, especially number of CPU cores and number/type of disks (default: 20)

standby_slaves_per_pool

Minimum number of standby slaves to keep in every pool (default: 2)

mysql_schema

database name (mandatory)

mysql_app_user

mysql user that your application uses (mandatory)

mysql_app_password

mysql password that your application uses (mandatory)

mysql_repl_user

mysql user for replication (mandatory)

mysql_repl_password

mysql password for replication (mandatory)

mysql_root_password

mysql root password (default: false, indicating that Jetpants should use /root/.my.cnf instead)

mysql_grant_ips

mysql user manipulations are applied to these IPs (array; mandatory)

mysql_grant_privs

mysql user manipulations grant this set of privileges by default (array; default: ['ALL'])

compress_with

command line to perform compression during large file copy operations; see below (default: false)

decompress_with

command line to perform decompression during large file copy operations; see below (default: false)

export_location

directory to use for data dumping (default: '/tmp')

verify_replication

raise exception if the actual replication topology differs from Jetpants' understanding of it (ie, disagreement between asset tracker and probed state), or if MySQL's two replication threads are in different states (one running and the other stopped) on a DB node. (default: true. master promotion tool ignores this, since the demoted master may legitimately be dead/offline)

private_interface

name of private interface on your servers, such as eth0 or bond0. Not used by any core Jetpants commands, but can be useful in plugings that wrap tcpdump, or code calling the Jetpants::Host.local method. (default: 'bond0')

plugins

hash of plugin name => arbitrary plugin data, usually a nested hash of settings (default: {})

ssh_keys

array of SSH private key file locations, if not using standard id_dsa or id_rsa. Passed directly to Net::SSH.start's :keys parameter (default: nil)

sharded_tables

array of name => {sharding_key=>X, chunks=>Y} hashes, describing all tables on shards. Required by shard split/rebuild processes (default: [])

Compression

Jetpants has the ability to use compression during large file copy operations, which are performed by commands “jetpants clone_slave” and “jetpants shard_split”. Compression is disabled by default in Jetpants unless you specify a compression program to use via the compress_with and decompress_with config options. It is highly recommended that you do so, in order to speed up these operations when working with large data sets.

The command lines that you specify should accept input from STDIN and supply output to STDOUT, because they will be used in the middle of a series of piped commands. The binary specified should be in root's PATH on all database nodes. We recommend use of a parallel compression tool, to take advantage of multiple cores.

You will need to do some profiling to determine the best tool to use for your hardware and data set; there's no universal best choice of compression algorithm or settings.

Some example values of these parameters are as follows:

Disable compression (default)

compress_with: false
decompress_with: false

pigz

pigz is an open-source parallel gzip tool by Mark Adler. It is available as a package in several Linux distros. It performs well, but is very CPU intensive. More information: zlib.net/pigz/

compress_with: pigz
decompress_with: pigz -d

qpress

qpress is a multi-threaded portable file archiver using QuickLZ. A prebuilt package is not available for most Linux distros due to licensing reasons, but a binary can be downloaded from www.quicklz.com/. It performs extremely well, especially once tuned.

In order to read from STDIN and write to STDOUT, use qpress -io. In this case qpress still requires a filename during compression, even though it is unused. Decompression does not have the same requirement.

The example below uses 4 threads and a block size of 32768KB.

compress_with: qpress -ioT4K32768 dummyfilename
decompress_with: qpress -dioT4

lzop

lzop is a less CPU-intensive compressor. lzop is still single-threaded in v1.x, so its performance may not be ideal for the Jetpants use-case. Multithreading is planned for v2.x. More information: www.lzop.org/

compress_with: lzop
decompress_with: lzop -d
Jump to Line
Something went wrong with that request. Please try again.