Metrics generation by the operator

[yorugac]

Kubernetes components come with metrics built-in, more or less, so the operator should try to follow the same principle. One goal is to monitor what tests are being run via the operator. Another goal would be to monitor the operator itself. Potential rabbit holes to dig in:

• kube-state-metrics CustomResourceStateMetrics
• is there a real alternative to this solution? What about limited types of metrics provided by CustomResourceStateMetrics, is there a workaround?

Some of the things a user of the operator might be interested in:

• counter for test runs, with label {test="foo"}
  • separately, counter for cloud output test runs or via additional label? (rationale: to count test runs subjected to billing)
  • counter for number of instances per {test="foo"}, e.g. in case if number of instances are set by devs / QAs and should be monitored. It might be of interest to separate "how many were configured in test" VS "how many were actually started".
• counter for successful and failed test runs
• gauge for total duration? E.g. to measure the bootstrapping delay vs duration of the test run.

Options for labels: test, k6-operator version, namespace, service account.