Storage Policy Model

Storage Policy Overview

Herd Storage Policy feature (planned Q4 2015 release) provides centralized management across storage tiers via meta-data. It allows Publishers to define a Storage Policy and then herd will execute that policy to move their data across different herd-managed Storage Platforms. The driving use case for this is to move data from S3 to Glacier to reduce storage costs. But it can also be used for other transitions in the future.

Storage Policy Model

The Storage Policy is made up of a Scope that defines the BData that the policy covers, a Rule that is evaluated to determine if a given BData should move, and a Transition that defines the source and destination Storage.

• Scope - expressed by limited subset of alternate keys and listed here in order of precedence
  • Always include StorageName of current storage location
  • BDef and (Usage & FileType) - override at this specific level
  • BDef - override at this specific level
  • (Usage & FileType) - a cross-cutting scope that applies to all BData of this Usage and FileType
  • [none] - global scope that applies to all BData
  • NOTE: (Usage & FileType) must always be used together to keep it simple - or else there are too many permutations
• Rule - rule type and numeric value
  • Initially support one rule type of DAYS_SINCE_BDATA_REGISTERED with the numeric value in days
• Transition - destination Storage (required)
  • Initially support only Glacier platform as destination

Storage Policy Examples

Below is an example is a global policy that applies to all BData in an S3 Bucket and says everything can be moved to Glacier after 7 years

StoragePolicy
  Name
Scope
  StorageName
Rule
  Type = DAYS_SINCE_BDATA_REGISTERED
  Value = 2555
Transition
  DestinationStorageName = SomeGlacierVault

Below is an example is a cross-cutting policy that applies to all BData of SRC Usage and TXT File Type as these can all be moved to Glacier after 90d

StoragePolicy
  Name
Scope
  StorageName
  Usage + FileType = SRC + TXT
Rule
  Type = DAYS_SINCE_BDATA_REGISTERED
  Value = 90
Transition
  DestinationStorageName = SomeGlacierVault

Below is an is an overriding Storage Policy which says all BData of a specific BDef need to be retained in S3 for 365 days and not have it's SRC/TXT Format archived in 90d by the cross-cutting policy

StoragePolicy
  Name
Scope
  StorageName
  BDef = Application_A/Definition_B
Rule
  Type = DAYS_SINCE_BDATA_REGISTERED
  Value = 365
Transition
  DestinationStorageName = SomeGlacierVault