Backups

Priam supports both snapshot and incremental backups for Cassandra SSTable files and uses S3 to save these files.

Snapshot or complete backup

Priam leverages Cassandra's snapshot feature to have an eventually consistant backup. Cassandra's snapshot feature flushes data to disk and hard links all SSTable files into a snapshot directory. These files are then picked up by Priam and uploaded to S3. Although snapshotting across cluster is not guaranteed to produce consistant backup of cluster, consistency is resumed upon restore by Cassandra and running repairs. Priam uses Quartz to schedule all tasks. Snapshot backups are run as a recurring task. They can also be triggered via REST API (refer API section), which is useful during upgrades or maintenance operations.
Snapshots are run on a daily basis for the entire cluster, at a specific time, Ideally during non-peak hours. Priam's backup.hour property allows you to set daily snapshot time (refer properties). The snapshot backup orchestrates invoking of Cassandra 'snapshot' JMX command, uploading files to S3 and cleaning up the directory.

meta.json

All SSTable files are uploaded individually to S3 with built in retries on failure. Upon completion, the meta file (meta.json) is uploaded which contains reference to all files that belong to the snapshot. This is then used for validation during restore.

Multi part upload

Priam leverages the multi part upload feature of S3. This helps in uploading large files in parallel. Each file is chunked into several parts and each part is uploaded in parallel.

Incremental backup

When incremental backups are enabled in Cassandra, hard links are created for all new SSTables created in the incremental backup directory. Since, SSTables are immutable files they can be safely copied to an external source. Priam scans this directory frequently for incremental SSTable files and uploads to S3.

Backup file path

To identify backup files and to avoid collision, Priam organizes the files in S3 by including time, keyspace name, token, date, DC/region and cluster name. Here is the general format:

<App Prefix>/<Region or DC>/<Cluster name>/<Token>/<Date & Time>/<File type>/<Keyspace>/<FileName>

Here are valid file types (refer AbstractBackupPath.BackupFileType):

SNAP - Snapshot file
META - Meta file for all snapshot files 
SST - Incremental SSTable file

Here is a sample snapshot file in S3:

test_backup/eu-west-1/cass_test/37809151880104273718152734159458104939/201202142346/SNAP/MyKeyspace/MyCF-g-265-Data.db

Compression

Priam uses snappy compression to compress data files. Each file is chunked and compressed on the fly.

Throttling

Throttling of uploads helps reduce disk IO and network IO as well. With Priam properties, you can set the Mb/s which are read from the disk.

Restore

Data from a snapshot can be restored to the same cluster or a different cluster. To restore a cluster, user specifies a start and end time for which the backup data is available. Priam executes the following sequence of steps for restoring:

• Search for the latest snapshot within the time range
• Download meta file for the snapshot
• Download snapshot files specified in the meta file
• Download all available incremental files upto the end time
• Start the cluster and join the ring The LocationInfo files are ignored and not backed up. This forces cassandra to rediscover other nodes in the cluster upon restore and refresh the ring nodes.

Restores can only be performed on the same size cluster or half size cluster. Restoring to half size cluster will skip alternate nodes and requires repair to be run to regain consistency.
Note that restoring to a different sized cluster is supported only for the Keyspaces with replication factor more than 1.

References

• More on Cassandra's backup feature on Datastax blog here.