METRON-392 Allow User to Define Custom 'Group By' for a Profile

[nickwallen]

METRON-392

Allows a user to optionally define a custom set of 'groupBy' expressions that controls how the data is persisted. This is intended to allow for contiguous scans when training on subsets of the data.

The 'groupBy' expressions can refer to any field within a ProfileMeasurement. This includes the following fields:

• profileName: The name of the profile.
• entity: The name of the entity being profiled.
• start: The window start time in milliseconds from the epoch.
• end: The window end time in milliseconds from the epoch.
• value: The summary value calculated over the window period.
• groupBy: The set of 'groupBy' expressions; not the result of those expressions.

A common use case would be grouping the data by day of week. This would allow a contiguous scan to access all profile data for Mondays only. The Stellar expression DAY_OF_WEEK(start) would achieve this.

NOTE: A series of date functions will be added to Stellar in a follow-on PR to enhance the types of groups that can be created.

Example

{
  "inputTopic": "indexing",
  "profiles": [
    {
      "profile": "example3",
      "foreach": "ip_src_addr",
      "onlyif": "protocol == 'HTTP'",
      "groupBy": "DAY_OF_WEEK(start)",
      "update": { "s": "STATS_ADD(s, length)" },
      "result": "STATS_MEAN(s)"
    }
  ]
}

Testing

To test this change do the following.

• Create a profile and do not define a 'groupBy' expression. Prior to this change the row key would include the day of week, week of month, etc which altered how the data was sorted on disk. After this change, these fields will not be included in the row key.
• Create a profile and define a 'groupBy' expression. The result of this expression will be embedded in the row key.

[nickwallen]

#231 includes the necessary Stellar date functions to handle the common uses cases of grouping by day of week, etc.

[cestella]

Why are you setting the salt divisor to 0? Under what situations would you not need a salt? I'm especially concerned that this appears to be the default behavior.

[nickwallen]

Restored default to 1000. Thanks

[nickwallen]

@cestella Updated the PR based on your feedback. Let me know what you think.

[cestella]

+1, looks good post master merge