Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
branch: master
Fetching contributors…

Cannot retrieve contributors at this time

118 lines (80 sloc) 5.734 kb

wordnik open-source tools

Overview

These are tools used to maintain a MongoDB deployment.

To build

Requires apache ant 1.7 or greater, java 6:

ant -f install-ivy # only needed if you don't have apache ivy installed
ant dist

To run

cd dist (or wherever you unzipped the distribution)
./bin/run.sh  

To get tool options

Run any tool with a -? parameter to see the options:

./bin/run.sh com.wordnik.system.mongodb.SnapshotUtil -?
usage: SnapshotUtil
 -c : CSV collection string (prefix with ! to exclude)
 -d : database name
 -h : hostname
 -t : threads
 -o : output directory
 [-s : max file size in MB]
 [-Z : compress files]
 [-J : output in JSON (default is BSON)]
 [-u : username]
 [-p : password]

Tools included

com.wordnik.system.mongodb.SnapshotUtil

This is pretty straight forward, it's meant for taking backups of your data. Some differences between it and mongodump are:

- It splits files based on a configurable size

- It will let you select what you want to backup with inclusion and exclusion operators

- It will automatically gzip the files as it rotates them

- It supports a JSON export

- It runs a configurable thread pool so you can backup multiple collections simultaneously

examples: backup localhost/test to folder backup

./bin/run.sh com.wordnik.system.mongodb.SnapshotUtil -h localhost -d test -o backup

backup only collection "users" in localhost/test to folder "backups"

./bin/run.sh com.wordnik.system.mongodb.SnapshotUtil -h localhost -d test -c users -o backups

backup collection "user_links" in localhost/test to folder backups in JSON format in 32mb files, then gzip

./bin/run.sh com.wordnik.system.mongodb.SnapshotUtil -h localhost -d test -c user_links -o backups -J -s 32 -Z

com.wordnik.system.mongodb.RestoreUtil

Operates against either mongodump files or files made with the SnapshotUtil with either uncompressed or compressed bson files. Also supports inclusion/exclusion of files

examples: Restore all files in folder "backups" to database "restored_data"

./bin/run.sh com.wordnik.system.mongodb.RestoreUtil -i backup -h localhost -d restored_data

com.wordnik.system.mongodb.IncrementalBackupUtil

This queries a master server's oplog and maintains a set of files which can be replayed against a snapshot of the database. The procedure we use is to snapshot the db (either at the filesystem or with the tool) and apply the incremental changes created by this tool.

The tool looks for a file called "last_timestamp.txt" in the output directory. This file sets a starting point for querying the oplog--it should contain a single line in the format:

[time-in-seconds]|[counter]

The [time-in-seconds] is the seconds since epoch, which you can grab from the OS or from a tool like this:

http://www.esqsoft.com/javascript_examples/date-to-epoch.htm

The counter should typically be set to 0. As the tool runs, every operation flushed to disk will cause this file to be updated. If you want to stop a running process, create a file called "stop.txt" in the CWD of the application. It will cause the app to stop within one second.

examples: Save incremental backup on everything in the database "ugc" (note the -c args are scoped to the database)

./bin/run.sh com.wordnik.system.mongodb.IncrementalBackupUtil -c ugc -o backups

Save incremental backup on the database "ugc" with collection "login_info" to the folder "backups"

./bin/run.sh com.wordnik.system.mongodb.IncrementalBackupUtil -c ugc.login_info -o backups

com.wordnik.system.mongodb.ReplayUtil

Takes a series of files created by the IncrementalBackupUtil and replays them. The tool allows applying the operations against alternate databases and collections. It also supports skipping records which fall outside a specified timepoint, if for instance you want to roll back to a particular point in time.

Note that not all operations can be remapped, especially adding indexes and applying database-level commands.

examples: Replay incremental backup files in folder "backups" to host "localhost:27018"

./bin/run.sh com.wordnik.system.mongodb.ReplayUtil -i backups -h localhost:27018

Replay files in folder "backups" to host "localhost:27018", collection "test1", and map from db "test" to db "foo". Note! This doesn't replay commands applied to remapped databases

./bin/run.sh com.wordnik.system.mongodb.ReplayUtil -i backups  -h localhost:27018 -R test=foo -c test1

Replay files in folder backups to host localhost:27018, collection test1, and map from test1 to test5

./bin/run.sh com.wordnik.system.mongodb.ReplayUtil -i backups  -h localhost:27018 -r test1=test5 -c test1

com.wordnik.system.mongodb.ReplicationUtil

Tool to replicate from server to server. Use caution with the config to avoid getting in a replication loop! You can add mappings to replicate from one database to another.

examples: Replicate everything from localhost:27017 to localhost:27018

./bin/run.sh com.wordnik.system.mongodb.ReplicationUtil -h localhost -H localhost:27018

Replicate everything from localhost:27017 to localhost:27018, and map from source database "foo" to target database "bar"

./bin/run.sh com.wordnik.system.mongodb.ReplicationUtil -h localhost -H localhost:27018 -m foo:bar

Replicate collection test.foo from localhost:27017 to localhost:27018, including commands, index operations

./bin/run.sh com.wordnik.system.mongodb.ReplicationUtil -h localhost -H localhost:27018 -c test.foo,test.$cmd,test.system.indexes
Jump to Line
Something went wrong with that request. Please try again.