2021.04.12

Storage Partitions Issues:

• Storing issues due to 10^7 on avg metrics per seconds
• Storage backups
• Variable performance of hierarchical storage.
• When trying to model or visualize historical data
• How to protect user data over time.

Use Cases:

• Take advantage of tiered storage performance:
  • New data goes to NVME storage

Containers:

• Contains schema, partitions, and configurations
• Distributes queries and collects results across partitions
• Implements iterators and maintains index order
• Multiple containers can share partitions

Partition Attributes:

• Per container:
• Name: identifier for partitions and is unique to a container
• State: Primary, Active, or Offline
• Global:
  • UUID: Universally Unique ID
  • Path: The storage location
  • Description: text to describe the data stored in the partition
  • Stats: storage bytes, modification times, access times, access rates, data rates, etc.
• States:
  • Primary:
    • All new data goes to this partition
  • Active:
    • Can see all data in the partition and can be queried
  • Offline:
    • Usually a temporary state before you move it. Cannot be seen.
• Operations:
  • Create
  • Attach
  • Set-State
  • Detach
  • Set State
  • Move
  • Query
  • Uses Py-API, C-API, or CMD.
• Schema:
  • Partitions contain objects
  • Every object has a schema to describe its data
  • The schema ID, therefore, must be unique across containers
    • 16B schema UUID on each object
    • Originally uint64_t
  • This is a bigger push towards handling different schema versions with the same name at the aggregator.
  • Going towards having a global database of schema UUID to tell the difference and the uniqueness of the schemas

Upcoming Additions (2021):

• Security and storage policies:
  • Object creation conditionals
    • Determines which partitions store the object
  • Storage strategies
    • Logs
• Storage conditionals:
  • Evaluate the object commit time
  • Determine which partition stores an object
  • Conditional statements:
    • Term: attr-type, attr-name
    • Boolean: or, and , not
    • Comparator: <, >, etc
  • Storage conditional goals:
    • Security policy
    • Support policy storing a user:group into partition
    • The target partition is protected by filesystem permissions
    • Container opened by one user, only the partition to which permission is granted is visible/writable
  • Storage management:
    • Add new objects to different partition based on data
    • Allows for single policy to be created and avoid race behavior when attempting to "roll-over" new data to the next partition
    • Older partitions may then be migrated to secondary storage without issues
  • Storage strategy:
    • Log structure storage

Other News:

• Several new LLNL samplers now available on OVIS-4
• AMD RDC GPU sampler now available on OVIS-4
• Pending pull-requests
  • New "ibmad" LLNL sampler
  • Procnet sampler