2022.01.31

• 4.3.8 is released

• SOS schema digest and UUID to track metric set schema and backend storage.

• Bug fixes and resiliency improvements

• New Features:

  • Multi-threaded low-level Zap transport event handler
  • Command line option support in config file
  • Summary set, transport, producer, and thread stats
  • Kokkos-appmon store
  • Darshan store
  • Non-blocking event lagging
  • Netlink notifier stream sampler

• Schema:

  • Schema names are administratively assigned. Unreliable for determining similarity of schema
  • Many schema are constructed from the /proc file from which the sampler collects data
  • Storage and analysis plugins need to know format of data they are handling
  • Plugins cache info about metric set schema and need to be able to determine if two sets are the same.
  • Does not affect backwards compatibility
  • LDMS set digest computed when schema is created
  • Includes a hash of all metric names and types
  • Does not include array lengths
  • Consumer can compute the hash to determine if the claimed digest is correct

• Use cases:

  • Determine if cache info about a metric set schema is correct
  • CSV store plugin: map digest to an output file and CSV header
  • Sos store plugin: map digest to a sos schema UUID and associated SOS schema
  • Stream store: map digest to stream name
  • Future: map digest to an LDMS row format config and split storage technology from storage policy.

• SOS schema UUID:

  • UUID computed using libuuid function. It is not computable from the schema definition
  • Every object in SOS partition has a header that contains the UUID that defines its type
  • In order to combine partitions gathered over time into a container for analysis a global database is required so that analysis can unambiguously qualify the data.

• Schema directory:

  • A schema directory is a JSON formatted file that defines a set of schema
  • There are commands to export and import schema directory into a SOS container
  • This allows for partitions to be combined into a container for the purpose of analysis
  • Flexible stores

• Storage config:

  • Maps a metric set into one or more rows in a data storage technology: SQL tables, so objects, CSV lines, JSON formats
  • Allows for config to define how sets are decomposed into rows.