[RFE] kube-burner comparisons

[rsevilla87]

Is your feature request related to a problem? Please describe.

By using the comparison library from go-commons we could provide kube-burner the capacity of comparing some of the collected metrics with previously indexed data in order to detect regressions automatically.

Describe the solution you'd like

One of the strategies to address this functionality can be to create an additional configuration file holding a comparison criteria, that can be broken down in a list of queries and tolerancies.
Given that the benchmarks launched from the OCP wrapper add some interesting metadata to all the indexed documents we can perform those comparisons against a subset of benchmarks rather than a single UUID.

Something like the below can be an initial approach.

- query: metadata.ocpMajorVersion.keyword: 4.14 AND metadata.platform.keyword: AWS AND metadata.sdnType.keyword: OVNKubernetes AND metadata.totalNodes: 125 AND metricName.keyword: podLatencyQuantilesMeasurement AND quantileName.keyword: Ready
  field: P99
  tolerancy: -25  # This means that the comparison will support up to 25% of latency increase
  stat: Avg # By default will be Avg, but we support more, listed at https://github.com/cloud-bulldozer/go-commons/blob/main/comparison/types.go#L20-L25
  metric: 
    metricName: podLatencyQuantilesMeasurement
    quantileName: Ready

# Next one uses 99thEtcdDiskWalFsync, which is a metric from metrics-report profile
- query: metadata.ocpMajorVersion.keyword: 4.14 AND metadata.platform.keyword: AWS AND metadata.sdnType.keyword: OVNKubernetes AND metadata.totalNodes: 125 AND metricName.keyword: 99thEtcdDiskWalFsync
  field: value
  tolerancy: -25  # This means that the comparison will support up to 25% of latency increase
  metric: 
    metricName: 99thEtcdDiskWalFsync

Where:

• query: is the Lucene query passed to https://github.com/cloud-bulldozer/go-commons/blob/main/comparison/elastic.go#L47, used to the document(s), we would like to compare the benchmark results with
• field: specifies the document's field we like to compare, defaults to value
• tolerancy: specifies the allowed tolerancy, can be a negative or a positive integer
• stat: The type of aggregation we want to compare the result against. By default Avg, more supported at https://github.com/cloud-bulldozer/go-commons/blob/main/comparison/elastic.go#L47
• metric: this is the criteria kube-burner will use to figure out which of the collected metrics will be compared. Is a dictionary of fields, setting only metricName should be enough normally

This is a preliminary idea, as usual I'm open to others thoughts & feedback.

cc: @jtaleric @dry923 @vishnuchalla @smalleni @afcollins

[vishnuchalla]

@rsevilla87 Why do we need this when we have grafana dashboards and a UI dashboard for doing similar kinds of aggregations?

[jtaleric]

@rsevilla87 Why do we need this when we have grafana dashboards and a UI dashboard for doing similar kinds of aggregations?

well the dashboards are great for humans. not so great to derive pass fail in an automated environment.

the intent is to use the comparison to determine if a run is good or bad based on our KPIs.

[github-actions]

This issue has become stale and will be closed automatically within 7 days.