Skip to content
Wordnik Open Source Software
Pull request Compare This branch is 182 commits behind reverb:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.
bin
src/main/java/com/wordnik
README.md
build.xml
install-ivy.xml
ivy.xml

README.md

wordnik open-source tools

Overview

These are tools used to maintain a MongoDB deployment.

To build

Requires apache ant 1.7 or greater, java 6:

ant -f install-ivy # only needed if you don't have apache ivy installed
ant dist

To run

cd dist (or wherever you unzipped the distribution)
./bin/run.sh  

To get tool options

Run any tool with a -? parameter to see the options:

./bin/run.sh com.wordnik.system.mongodb.SnapshotUtil -?
usage: SnapshotUtil
 -c : CSV collection string (prefix with ! to exclude)
 -d : database name
 -h : hostname
 -t : threads
 -o : output directory
 [-s : max file size in MB]
 [-Z : compress files]
 [-J : output in JSON (default is BSON)]
 [-u : username]
 [-p : password]

Tools included

com.wordnik.system.mongodb.SnapshotUtil

This is pretty straight forward, it's meant for taking backups of your data. Some differences between it and mongodump are:

  • - It splits files based on a configurable size
  • - It will let you select what you want to backup with inclusion and exclusion operators
  • - It will automatically gzip the files as it rotates them
  • - It supports a JSON export
  • - It runs a configurable thread pool so you can backup multiple collections simultaneously
  • examples: backup localhost/test to folder backup

    ./bin/run.sh com.wordnik.system.mongodb.SnapshotUtil -h localhost -d test -o backup

    backup only collection "users" in localhost/test to folder "backups"

    ./bin/run.sh com.wordnik.system.mongodb.SnapshotUtil -h localhost -d test -c users -o backups

    backup collection "user_links" in localhost/test to folder backups in JSON format in 32mb files, then gzip

    ./bin/run.sh com.wordnik.system.mongodb.SnapshotUtil -h localhost -d test -c user_links -o backups -J -s 32 -Z

    com.wordnik.system.mongodb.RestoreUtil

    Operates against either mongodump files or files made with the SnapshotUtil with either uncompressed or compressed bson files. Also supports inclusion/exclusion of files

    examples: Restore all files in folder "backups" to database "restored_data"

    ./bin/run.sh com.wordnik.system.mongodb.RestoreUtil -i backup -h localhost -d restored_data

    com.wordnik.system.mongodb.IncrementalBackupUtil

    This queries a master server's oplog and maintains a set of files which can be replayed against a snapshot of the database. The procedure we use is to snapshot the db (either at the filesystem or with the tool) and apply the incremental changes created by this tool.

    The tool looks for a file called "last_timestamp.txt" in the output directory. This file sets a starting point for querying the oplog--it should contain a single line in the format:

    [time-in-seconds]|[counter]

    The [time-in-seconds] is the seconds since epoch, which you can grab from the OS or from a tool like this:

    http://www.esqsoft.com/javascript_examples/date-to-epoch.htm

    The counter should typically be set to 0. As the tool runs, every operation flushed to disk will cause this file to be updated. If you want to stop a running process, create a file called "stop.txt" in the CWD of the application. It will cause the app to stop within one second.

    examples: Save incremental backup on everything in the database "ugc" (note the -c args are scoped to the database)

    ./bin/run.sh com.wordnik.system.mongodb.IncrementalBackupUtil -c ugc -o backups

    Save incremental backup on the database "ugc" with collection "login_info" to the folder "backups"

    ./bin/run.sh com.wordnik.system.mongodb.IncrementalBackupUtil -c ugc.login_info -o backups

    com.wordnik.system.mongodb.ReplayUtil

    Takes a series of files created by the IncrementalBackupUtil and replays them. The tool allows applying the operations against alternate databases and collections. It also supports skipping records which fall outside a specified timepoint, if for instance you want to roll back to a particular point in time.

    Note that not all operations can be remapped, especially adding indexes and applying database-level commands.

    examples: Replay incremental backup files in folder "backups" to host "localhost:27018"

    ./bin/run.sh com.wordnik.system.mongodb.ReplayUtil -i backups -h localhost:27018

    Replay files in folder "backups" to host "localhost:27018", collection "test1", and map from db "test" to db "foo". Note! This doesn't replay commands applied to remapped databases

    ./bin/run.sh com.wordnik.system.mongodb.ReplayUtil -i backups  -h localhost:27018 -R test=foo -c test1

    Replay files in folder backups to host localhost:27018, collection test1, and map from test1 to test5

    ./bin/run.sh com.wordnik.system.mongodb.ReplayUtil -i backups  -h localhost:27018 -r test1=test5 -c test1

    com.wordnik.system.mongodb.ReplicationUtil

    Tool to replicate from server to server. Use caution with the config to avoid getting in a replication loop! You can add mappings to replicate from one database to another.

    examples: Replicate everything from localhost:27017 to localhost:27018

    ./bin/run.sh com.wordnik.system.mongodb.ReplicationUtil -h localhost -H localhost:27018

    Replicate everything from localhost:27017 to localhost:27018, and map from source database "foo" to target database "bar"

    ./bin/run.sh com.wordnik.system.mongodb.ReplicationUtil -h localhost -H localhost:27018 -m foo:bar

    Replicate collection test.foo from localhost:27017 to localhost:27018, including commands, index operations

    ./bin/run.sh com.wordnik.system.mongodb.ReplicationUtil -h localhost -H localhost:27018 -c test.foo,test.$cmd,test.system.indexes
    Something went wrong with that request. Please try again.