jsonnet/telemeter: Record a "subscribed cluster" metric #297
Conversation
We have defined the OpenShift SLI based on whether the cluster version operator is reporting a failure or not over a period of time. To better support queries that attempt to aggregate availability over classes of cluster, create a recording rule that is 1 when a cluster is available and 0 when the cluster is reporting unhealthy. If the cluster is not reporting any metrics it would have no data at that point in time. Include the current version of the cluster since it is implicit at the time of the recording rule. This can be used in more complex aggregates by joining on cluster id.
There are a number of key dimensions joined against the subscription data that almost all queries base on. 1. Is it subscribed 2. Is it internal 3. Is it openshift dedicated 4. Is it in a range of versions 5. AND is it currently running (subscription_labels does not check that) This recording rule collapses those into a single metric with cardinality O(clusters) that should dramatically reduce the cost of queries. Combined with the previous rule (available) the extremely common query "which subscribed clusters are unhealthy" becomes ``` id_version:cluster_available == 0 + on(_id) max by (_id) (id_version_ebs_account_internal:cluster_subscribed == 1) ``` This also reduces the data that must be searched for almost all other joins and some dashboards.
openshift-ci-robot
added
approved
|
/retest /assign @brancz |
|
@smarterclayton: GitHub didn't allow me to assign the following users: fbrancz. Note that only openshift members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/retest |
5 similar comments
|
/retest |
|
/retest |
|
/retest |
|
/retest |
|
/retest |
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: brancz, smarterclayton The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
There are a number of key dimensions joined against the subscription data
that almost all queries base on.
This recording rule collapses those into a single metric with cardinality
O(clusters) that should dramatically reduce the cost of queries.
Combined with the previous rule (available) the extremely common query
"which subscribed clusters are unhealthy" becomes
This also reduces the data that must be searched for almost all other joins
and some dashboards.
Question: how expensive is this recording rule?