Skip to content

derfshaya/rocks-strata

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

47 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

rocks-strata: A Framework for Managing Incremental Backups with the RocksDB Storage Engine

Build Status

rocks-strata is a framework for managing incremental backups of databases that use the RocksDB storage engine. Current drivers support MongoDB with the RocksDB storage engine ("MongoRocks") and use Amazon S3 for remote storage. rocks-strata also includes a tool to query backups from a MongoDB shell without doing a restore.

rocks-strata takes advantage of RocksDB's architecture, which makes cheap incremental backups possible. It can be extended to work with different database management systems and different storage environments.

rocks-strata has been running in production at Parse since about August 13, 2015. Here's the blog post: http://blog.parse.com/learn/engineering/strata-open-source-library-for-efficient-mongodb-backups/

For more information about running your Mongo database with the RocksDB storage engine, see https://github.com/mongodb-partners/mongo-rocks.

Supported Platforms

rocks-strata should work on most Linux distributions. It has been tested on Ubuntu.

Scripts under examples/ might use tools that are not available on all distributions.

Getting Started

If you'd like to use rocks-strata with MongoRocks (which is the main focus of this project), you'll need to set up MongoRocks first: https://github.com/mongodb-partners/mongo-rocks.

You'll need to have Go installed to build rocks-strata. See https://golang.org/doc/install.

Get rocks-strata and its dependencies with go get github.com/facebookgo/rocks-strata/strata.

To back up up to S3, first build the driver:

cd rocks-strata/strata/cmd/lreplica_s3storage_driver/strata
go build

Check out ./strata --help. To see more about a subcommand, such as gc, do ./strata gc --help.

Try a one-off backup and restore

This driver looks for the environment variables AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID. So, do something like the following. Note: This example will write your AWS credentials to your bash history.

echo "export AWS_SECRET_ACCESS_KEY=REPLACEME" >> aws_credentials
echo "export AWS_ACCESS_KEY_ID=REPLACEME" >> aws_credentials
source aws_credentials

To try a one-off backup, create a test database.

mkdir tmpdb
mongod --dbpath=/absolute/path/to/tmpdb --storageEngine=rocksdb
# Change shells
mongo --eval 'db.testc.insert( { _id: "cheetah" } )'

Then, while mongod is still running, perform a backup! The values of --bucket, --bucket-prefix, and --replica-id below are just suggestions.

cd rocks-strata/strata/cmd/lreplica_s3storage_driver/strata
./strata --bucket=all_backups --bucket-prefix=mongo-rocks backup --replica-id=one-off-test

If you used a non-default port to start mongod, you'll need to specify it with ./strata --port=.

Run ./strata --bucket=all_backups --bucket-prefix=mongo-rocks show backups --replica-id=one-off-test to list the backups on S3. You should see something like this:

ID   data                      num files   size (GB)   incremental files   incremental size   duration
0    2015-09-02 21:11:20 UTC   4           0.000005    4                   0.000005           187.929573ms

Now, suppose your database server burns to the ground:

# kill mongod
rm -rf tmpdb

Restore and restart mongod:

./strata --bucket=all_backups --bucket-prefix=mongo-rocks restore --replica-id=one-off-test --backup-id=0 --target-path=/absolute/path/to/tmpdb
mongod --dbpath=/absolute/path/to/tmpdb --storageEngine=rocksdb

In another shell, launch the MongoDB shell (mongo) and enter db.testc.find(). You should see the record { "_id" : "cheetah" }.

Query backups from an extended MongoDB shell

Inspecting backups is almost as easy as inspecting live databases. First, build an extended MongoDB shell:

cd rocks-strata/strata/cmd/lreplica_s3storage_driver/mongoq
go build

This is a wrapper that adds new commands to the MongoDB shell. Check out ./mongoq --help.

To use this, you need to mount part of S3 as a read-only file system in user space. Set up a FUSE implementation such as yas3fs: https://github.com/danilop/yas3fs.

With yas3fs and the example bucket/prefix names from the previous section, run yas3fs s3://all_backups/mongo-rocks /path/to/desired/mountpoint --mkdir --no-metadata --read-only -f -d. You'll still need the AWS credentials environment variables from the previous section. If you need to use sudo, use sudo -E to preserve the environment variables.

Congrats, you should be set up to query arbitrary backups from a command-line shell! To make things a little more interesting, let's do another one-off backup first.

mongod --dbpath=/absolute/path/to/tmpdb --storageEngine=rocksdb
# Change shells
mongo --eval 'db.testc.insert( { _id: "tiger" } )'
cd rocks-strata/strata/cmd/lreplica_s3storage_driver/strata
./strata --bucket=all_backups --bucket-prefix=mongo-rocks backup --replica-id=one-off-test

Now, launch the extended MongoDB shell with ./mongoq -b=all_backups -p=mongo-rocks -m=/path/to/desired/mountpoint. If mongod and mongo aren't on your path, you'll need to use the --mongod-path and --mongo-path arguments as well. You should be able to go through a session like the one below. Some output has been replaced by "..." for brevity.

> help
...
> status
Current replica ID: ?
Current backup ID: ?
> lr
Replicas with backups in remote storage:
one-off-test
> cr one-off-test
Replica ID changing to one-off-test
Database state is undefined until change_backup [cb] is run
> lb
ID   data                      num files   size (GB)   incremental files   incremental size   duration
0    2015-09-02 21:11:20 UTC   4           0.000005    4                   0.000005           187.929573ms   
1    2015-09-03 16:00:06 UTC   6           0.000009    4                   0.000004           299.54062ms    
> cb 0
...
> show dbs
local  0.000GB
test   0.000GB
> db.testc.find()
{ "_id" : "cheetah" }
> cb 1
...
> db.testc.find()
{ "_id" : "cheetah" }
{ "_id" : "tiger" }
> exit

Set up regular backups

See examples/backups for ideas.

Note that for a given replica ID, at most one operation that is capable of writing to storage (backup, delete, gc) should be running at any one time. That is why examples/backups uses start-stop-daemon. It is safe to restore while a write-capable operation is underway, as long as the backup being restored from isn't deleted while the restore is in progress. Operations that use one replica ID do not affect operations that use a different replica ID.

To avoid impacting performance in production, you may wish to dedicate a hidden MongoDB replica to performing backups.

Set up monitoring

See examples/monitoring for ideas

Using password auth

If your mongo distribution has authentication enabled, you must specify a username and password when running the backup command. This user needs access to run the getCmdLineOpts and setParameter commands on the admin database. To configure a backup user with minimal access needed to run strata backups, you can define a custom role with only these permissions:

> db.createRole({role: "rocksBackupRole", privileges: [ { resource: {cluster: true}, actions: ["getCmdLineOpts", "setParameter"]}], roles: [] })
{
        "role" : "rocksBackupRole",
        "privileges" : [
                {
                        "resource" : {
                                "cluster" : true
                        },
                        "actions" : [
                                "getCmdLineOpts",
                                "setParameter"
                        ]
                }
        ],
        "roles" : [ ]
}
> db.createUser({user: "backupUser", pwd: "abc123", roles: [ {role: "rocksBackupRole", db: "admin"} ]})
Successfully added user: {
        "user" : "backupUser",
        "roles" : [
                {
                        "role" : "rocksBackupRole",
                        "db" : "admin"
                }
        ]
}

Extending Strata

You can extend rocks-strata by implementing the Storage and Replica interfaces, defined in manager.go. And, you will probably want to implement a driver similar to rocks-strata/strata/cmd/lreplica_s3storage_driver/strata/main.go.

About

No description, website, or topics provided.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Go 100.0%