Store deduplicated EBS snapshots to S3 for greater long term cost savings
Java CSS JavaScript HTML CoffeeScript Shell Other
Latest commit 2c74bee Jan 23, 2017 @rabay108 rabay108 committed on GitHub Merge pull request #146 from SungardAS/readme_update
IAM role link fix
Permalink
Failed to load latest commit information.
WebContent fix typing error with <Single Sign On> Jan 4, 2017
ansible removing copycat from ansible Nov 2, 2016
lib/CopyCat/CopyCat/0.0.1
resources
src/com/sungardas
test test ignore Nov 3, 2016
.gitignore SNAP-396: Oct 24, 2016
.lgtm
.project Added build instruction Aug 19, 2015
.tern-project Initial commit with current version of java code Jul 27, 2015
.travis.yml version fix Nov 7, 2016
BuildDeploy.md
ConfigurationManagement.md [SNAP-578] - Update documentation Jan 19, 2017
LICENSE.md
MAINTAINERS
README.md Merge fix Jan 23, 2017
SetupDevEnv.md
SetupEmailNotification.md style update Sep 21, 2016
Vagrantfile [SNAP-188] updated provision configuration for implementing reverse p… Sep 10, 2015
ami_deploy.sh
autobuild.md
autobuild.sh market_place_mar_2016 fixes Apr 7, 2016
pom.xml
snapdirector_ami.json removing copycat from ansible Nov 2, 2016
sungardas_es_admin_role_perm_restricted.template Update sungardas_es_admin_role_perm_restricted.template Apr 14, 2016

README.md

Enhanced Snapshots

Enhanced Snapshots Logo

Version Build Status

Table of contents

Product Description

Enhanced Snapshots from Sungard Availability Services | Labs manages Amazon EBS snapshots and performs data deduplication to Amazon Simple Storage (Amazon S3). Users of Amazon We Services (AWS) can leverage the product to:

  • Reduce the cost of storing snapshots
  • Reduce the time that IT engineers spend on routine snapshot management tasks
  • Schedule recurring snapshots

Deduplication is run across all enabled snapshots in the AWS region, thus decreasing the amount of the total stored data. Therefore, the resulting deduplicated blocks are stored in Amazon S3 at a much lower cost than the standard AWS EBS snapshots and you pay less for the long-term retention of the snapshot-based data in AWS.

Using an intuitive interface, you can easily automate such routine tasks like creating snapshots or deleting old backups. Since these tasks are automated, risks associated with the human error are minimized.

Technical support is available only starting with Enhanced Snapshots 2.0.0. If you have any comments or suggestions, you can create a Github issue. Customer support may be added in a future release.

Enhanced Snapshots is open sourced and licensed under the Apache License 2.0. Use of the Enhanced Snapshots software is free. You pay only for the underlying infrastructure that is required to support it.

The Enhanced Snapshots tool is available after launching the Amazon Machine Image (AMI) from the enhanced snapshots market place by selecting the Enhanced Snapshots tool creation stack option or by creating the es-admin role while using the single AMI option. You can use the cloud formation template mentioned in the Quick start section to create the es-admin role. Similarly, you can uninstall the created EC2 instance and associated resources. However, after performing the uninstallation, you will not be removed from the AWS subscription for the SungardAS provided marketplace products. For more details, refer to Removing the enhanced snapshots system.

Key Features

Backup & Recovery

  • Create backups of the EBS volumes.
  • Perform instance recovery from the historical backups.
  • Store backups of the deleted volumes.
  • Quickly initiate the backups recovery.
  • Run backup and recovery tasks in parallel.

Schedule Policy

  • Scheduled tasks by minute, hour, day, week, and month.
  • Full support of the CRON expressions.

Retention Policy

  • Automatically delete older backups based on the original volume size, count, or age.

Management

  • Simple and intuitive wizard for initial setup process.
  • Admin and User roles.
  • Migration to another EC2 instance with full data restoration.
  • Single sign-on. For more information on how to configure single sign-on, see the Configuration Management file.
  • Configure and view runtime logs.
  • View task history.

Limitations

  • No support for managing the volumes that use the OS level RAID.
  • To avoid a significant storage overhead, the EBS volumes, which use the EBS encryption, must be pre-warmed.

Quick start

  • If you are using Enhanced Snapshots tool creation stack in the marketplace, the es-admin role is automatically created. For a single AMI option from the marketplace, create the es-admin role using the cloud formation template as prerequisite es-admin-role To launch a CloudFormation stack based on the template, choose a region in which you will perform the deployments. In this region, you will need the following information:

    • EC2 keypair.
    • VPC ID (not required if you are using the Simple Stack option below).
    • IP prefix from which the inbound HTTP and HTTPS connections are allowed. If you are not sure, use your current public IP address with /32 at the end (for example, 1.2.3.4/32).
    • IP prefix from which the inbound SSH connections are allowed. If you are not sure, use your current public IP address with /32 at the end (for example, 1.2.3.4/32).
  • Size of the instance EBS volume (in GB) should be calculated based on the 4% of the expected backup data size. For example, for storing 10 TB of data, the instance EBS volume should be 400 GB.

Note: When creating the CloudFormation stack, at the bottom of the Review page, select the check box in the Capabilities section.

Capabilities_checkbox

Once you have collected that information, find your target region, note the AMI ID for that region, and click the corresponding Launch Stack link.

Region AMI ID Simple Stack (default VPC) VPC Stack (your VPC)
us-east-1 ami-3f618429 Launch Stack Launch Stack
us-west-1 ami-6fa7f70f Launch Stack Launch Stack
us-west-2 ami-01c97a61 Launch Stack Launch Stack
eu-west-1 ami-a092bcd3 Launch Stack Launch Stack
eu-central-1 ami-f1ac619e Launch Stack Launch Stack
sa-east-1 ami-ba21bad6 Launch Stack Launch Stack
ap-southeast-1 ami-2b8f2748 Launch Stack Launch Stack
ap-southeast-2 ami-25111446 Launch Stack Launch Stack
ap-northeast-1 ami-8a8cfced Launch Stack Launch Stack
ap-northeast-2 ami-5b5c8a35 Launch Stack Launch Stack

Once the CloudFormation stack is built, go to its Outputs tab at the bottom of the AWS Console. Copy the instance ID (you will need it for logging in to the Enhanced Snapshots), click the URL, and then proceed to Getting Started.

You should also:

  • Select minimum m3.large instance and minimum 8 GB size for the volume.
  • Select the es_admin role while launching the instance to get access to CloudWatch.

Role

Getting Started

If you have not followed the Quick start section, then you will first need to manually create an IAM role, and then create an EC2 instance using the Enhanced Snapshots AMI, which can be found in the first table above.

Note: By default a new instance has a self-signed SSL certificate, so you need to bypass your browser security warning to start.

Step 1

To log in for the first time, use the following credentials:

  • Login: admin@enhancedsnapshots
  • Password: Your AWS EC2 Instance ID (available on the AWS console) Login

Step 2

After you log in, you can do the following:

  • Edit the S3 Bucket name. The S3 Bucket name must start with enhancedsnapshots and meet the general AWS requirements for the bucket names. Note: If you edit the S3 Bucket name on this page, you cannot modify it later.
  • Edit SDFS settings.
  • Enable Sign-on options. For more information on how to configure the single sign-on options, see the Configuration Management file.
  • Edit Email notifications. For more information on how to edit email notifications, see the Configuration Management file. Configuration Settings

Step 3

To create the first user that will automatically receive the administrator privileges, in the New User dialog box, specify the necessary information, and then click Add User.

Notes:

  • Email is used as the user ID when logging in to the application.
  • Password must not be less than eight characters and include at least one uppercase, one lowercase, one non-alphanumeric character, and one digit (from 0 through 9).

    New user

Step 4

After you click Add User, the system automatically creates an environment, and then redirects you to the new user login page.

To log in as a new user, type the credentials that you specified in step 3.

Login

After logging in as a new user, a list of the EBS volumes for the local region is displayed.

Volumes

Management Tasks

Creating Manual Backups

To create a backup:

  1. On the Volumes tab, select the appropriate EBS volumes, and then click Backup selected.

    Volumes (backup selected)

  2. In the Backup Volume dialog box, click Add backup task.

  3. In the confirmation dialog box, click either Go to Tasks or Stay on Volumes.

    Note: Only up to five backup tasks can be run in parallel. multiple backups

To cancel a backup task in process or delete a task from the history:

  1. On the Tasks tab, hover over the required task, and then click Reject.
  2. In the confirmation dialog box, click Reject or Delete.

To delete an old backup manually:

  1. On the Volumes tab, hover over the appropriate EBS volume, and then click Backup history.
  2. In the backup history list, select the appropriate volume backups, and then click Delete selected.
  3. In the confirmation dialog box, click Delete.

Restoring backups

To restore the latest backup: 1. On the Volumes tab, select the appropriate EBS volumes, and then click Restore selected.
2. In the Restore Backup dialog box, edit the availability zone to which the backup will be restored if necessary, and then click Add restore task. multiple restores 3. In the confirmation dialog box, click either Go to Tasks or Stay on Volumes. Note: Only up to five recovery tasks can be run in parallel.

To restore a backup from history: 1. On the Volumes tab, hover over the appropriate EBS volume, and then click Backup history.
2. In the backup history list, hover over the appropriate volume backup, and then click Restore. 3. In the Restore Backup dialog box, edit the availability zone to which the backup will be restored if necessary, and then click Add Restore Task. 4. In the confirmation dialog box, click either Go to Tasks or Stay on Volumes.

To cancel a recovery task in process or delete a task from the history:

  1. On the Tasks tab, hover over the required task, and then click Reject.
  2. In the confirmation dialog box, click Reject or Delete.

Creating a Schedule

You can automate the process of creating backups by scheduling the backup tasks. Schedules are displayed and stored in the Cron format. The interval between backups can be from one minute to one year. If necessary, schedules can be disabled.

To schedule a backup task:

  1. On the Volumes tab, hover over the appropriate volume, click Schedule, and then click New Schedule. Schedule button
  2. In the New Schedule dialog box, specify the name, frequency, and time for the schedule, ensure that the Enabled check box is selected, and then click Add Schedule.

New schedule

To view the scheduled tasks for a specific volume, on the Volumes tab, hover over the appropriate volume, and then click Schedule.

To edit a schedule:

  1. On the Volumes tab, click the appropriate volume.
  2. Hover over the required schedule for the volume, and then click Edit.
  3. Make the required modifications, and then click Save Schedule. Edit schedule

To delete a schedule:

  1. On the Volumes tab, hover over the appropriate volume, and then click Schedule.
  2. Hover over the required schedule for the volume, and then click Delete.

Managing Retention Policies

Using the retention policy, you can automatically delete backups according to their backup size, age, and number of containing files.

To create a retention rule:

  1. On the Volumes tab, hover over the appropriate volume, and then click Retention.

    Note: You can create only one retention rule per each volume that was previously backed up.

  2. In the Edit Retention Rule dialog box, do the following:

    • To delete a backup when its size exceeds a certain value, click the Size Limit tab, and then, in the backup size exceeds box, specify the limit.
      • To delete a backup when the number of files in this backup exceeds a certain value, click the Count Limit tab, and then, in the total files count exceeds box, specify the limit.
      • To delete a backup when it is older than a certain number of days, click the Days Limit tab, and then, in the backup creation date is over box, specify the limit.

    Edit retention rule

  3. Click Save Rule.

Filtering Volumes

You can quickly locate the required volumes by filtering them according to different parameters.

To filter the volumes:

  1. In the top right corner under your user name, click Filter.
  2. In the Filter Volumes dialog box, specify the appropriate filtering parameters, and then click Apply filter. Filter

Viewing Logs

You can view and trace the logs for all processes in the Enhanced Snapshots tool on the Logs tab. To trace the runtime logs, click Follow Logs.

Note: Only users with the administrator rights can access the Logs tab.

Managing System Settings

Depending on your requirements and goals, you can configure different system setting in the Enhanced Snapshots tool. For more information, see the Configuration Management file.

Managing System Migration

You can migrate to another EC2 instance without any loss of data by doing the following:

  • Ensure that no backup tasks are running
  • Perform a system backup. Go to the Setting tab, and then click Backup now.
  • Remember the created bucket name.
  • Uninstall the current system as described in Removing the Enhanced Snapshots system but do not remove the S3 bucket. Delete

  • Start a new instance with the same or newer version of Enhanced Snapshots.

  • During initial configuration step, in the S3 Bucket list, select the S3 bucket from the previous system.

Other Management Tasks

To view all active and pending tasks, go to the Tasks tab.

To specify the time limit for storing the tasks, go to Settings, and then, in the Tasks History Time Limit box, type the required value.

To view a list of all users, go to the Users tab.

If you have the administrator rights, you can edit information about all users including other Enhanced Snapshots administrators.

If you are a user without administrative rights, you can edit only your user profile.

If only one administrator user was created in the system, the administrator rights of this user cannot be revoked.

Removing the Enhanced Snapshots system

To uninstall Enhanced Snapshots, do the following:

  1. On the Settings tab, click Uninstall. Settings
  2. In the System Uninstall dialog box, do the following, and then click Uninstall.
    • Specify whether you want to remove the S3 Bucket.
    • Type your system ID.

System uninstall with remove bucket

Note: It can take several minutes to delete all resources, especially if you have stored the backup data.

The system continues with the removal of all resources once you enter the EC2 Instance ID for the EC2 instance on which the Enhanced Snapshots tool is running. The following resources are deleted:

  • EC2 Instance
  • S3 bucket and all the backup data
  • DynamoDB tables

Note: After the EC2 instance is deleted, the subscription to the software product from AWS is not removed. For more information, go to AWS Marketplace Help and FAQ.

IAM role creation (optional)

If you create an instance directly from the AMI without using the provided CloudFormation template, you must first create an IAM role with the policy as defined in the es-admin role cloud formation template. After the role is created, create and save an API key, which you will need to configure the Enhanced Snapshots tool.

Without a properly configured role, the following error message appears during the configuration: DynamoDBAccessDenied

Logging

Enhances Snapshots uses AWS CloudWatch as logs storage. You can find the logs at the following location: Logs

License

For license rights and limitations (Apache 2), see the LICENSE.md file.

Use of the provided AMIs is covered by a separate End User License Agreement.

Sungard Availability Services | Labs

Sungard Availability Services | Labs

This project is maintained by the Labs team at Sungard Availability Services

GitHub: https://sungardas.github.io

Blog: http://blog.sungardas.com/CTOLabs/