Bug 1992507: Use prometheus rule annotations comply with the OpenShift alerting guidelines

[creydr]

This PR switches to the OpenShift comply annotations for PrometheusRules:
The guidelines can be found at https://github.com/openshift/enhancements/blob/master/enhancements/monitoring/alerting-consistency.md#documentation-required

How to validate:

• start a cluster with this patch
• check the prometheus rules (e.g. for openshift-sdn): k -n openshift-sdn get prometheusrules -o yaml. e.g.:

$ k -n openshift-sdn get prometheusrules -o yaml
apiVersion: v1
items:
- apiVersion: monitoring.coreos.com/v1
  kind: PrometheusRule
  metadata:
    annotations:
      networkoperator.openshift.io/ignore-errors: ""
    creationTimestamp: "2021-08-12T09:05:57Z"
    generation: 1
    labels:
      prometheus: k8s
      role: alert-rules
    name: networking-rules
    namespace: openshift-sdn
    ownerReferences:
    - apiVersion: operator.openshift.io/v1
      blockOwnerDeletion: true
      controller: true
      kind: Network
      name: cluster
      uid: 29b2fb5e-e6d3-4b94-86d0-1ee64c104f16
    resourceVersion: "3441"
    uid: 56d32d96-c2a7-46df-9c29-2e325c343bc1
  spec:
    groups:
    - name: cluster-network-operator-sdn.rules
      rules:
      - alert: NodeWithoutSDNPod
        annotations:
          summary: All nodes should be running an sdn pod, {{ $labels.node }} is not.
        expr: |
          (kube_node_info unless on(node) topk by (node) (1, kube_pod_info{namespace="openshift-sdn",  pod=~"sdn.*"})) > 0
        for: 10m
        labels:
          severity: warning
      - alert: NodeProxyApplySlow
        annotations:
          summary: SDN pod {{ $labels.pod }} on node {{ $labels.node }} is taking
            too long, on average, to apply kubernetes service rules to iptables.
        expr: "histogram_quantile(.95, kubeproxy_sync_proxy_rules_duration_seconds_bucket)
          \n* on(namespace, pod) group_right topk by (namespace, pod) (1, kube_pod_info{namespace=\"openshift-sdn\",
          \ pod=~\"sdn-[^-]*\"}) > 15\n"
        labels:
          severity: warning
      - alert: ClusterProxyApplySlow
        annotations:
          summary: The cluster is taking too long, on average, to apply kubernetes
            service rules to iptables.
        expr: |
          histogram_quantile(0.95, sum(rate(kubeproxy_sync_proxy_rules_duration_seconds_bucket[5m])) by (le)) > 10
        labels:
          severity: warning
      - alert: NodeProxyApplyStale
        annotations:
          summary: SDN pod {{ $labels.pod }} on node {{ $labels.node }} has stale
            kubernetes service rules in iptables.
        expr: |
          (kubeproxy_sync_proxy_rules_last_queued_timestamp_seconds - kubeproxy_sync_proxy_rules_last_timestamp_seconds)
          * on(namespace, pod) group_right() topk by (namespace, pod) (1, kube_pod_info{namespace="openshift-sdn",pod=~"sdn-[^-]*"})
          > 30
        for: 5m
        labels:
          severity: warning
      - alert: SDNPodNotReady
        annotations:
          summary: SDN pod {{ $labels.pod }} on node {{ $labels.node }} is not ready.
        expr: |
          kube_pod_status_ready{namespace='openshift-sdn', condition='true'} == 0
        for: 10m
        labels:
          severity: warning
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Notes for reviewers:

• BZ 1992507 only complains about openshift-sdn. This bug adjusts the annotations for ovn & kuryr as well

[openshift-ci]

@creydr: This pull request references Bugzilla bug 1992507, which is invalid:

• expected the bug to target the "4.9.0" release, but it targets "---" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Bug 1992507: Use prometheus rule annotations comply with the OpenShift alerting guidelines

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[openshift-ci]

@creydr: This pull request references Bugzilla bug 1992507, which is invalid:

• expected the bug to target the "4.9.0" release, but it targets "---" instead
• expected Bugzilla bug 1992507 to depend on a bug targeting a release in 4.10.0 and in one of the following states: MODIFIED, ON_QA, VERIFIED, but no dependents were found

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Bug 1992507: Use prometheus rule annotations comply with the OpenShift alerting guidelines

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci]

@creydr: This pull request references Bugzilla bug 1992507, which is invalid:

• expected the bug to target the "4.9.0" release, but it targets "---" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Bug 1992507: Use prometheus rule annotations comply with the OpenShift alerting guidelines

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[creydr]

/bugzilla refresh

[openshift-ci]

@creydr: This pull request references Bugzilla bug 1992507, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target release (4.9.0) matches configured target release for branch (4.9.0)
• bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @zhaozhanqi

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci]

@creydr: This pull request references Bugzilla bug 1992507, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target release (4.9.0) matches configured target release for branch (4.9.0)
• bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @zhaozhanqi

In response to this:

Bug 1992507: Use prometheus rule annotations comply with the OpenShift alerting guidelines

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[creydr]

/test all

[openshift-ci]

@creydr: This pull request references Bugzilla bug 1992507, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target release (4.9.0) matches configured target release for branch (4.9.0)
• bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @zhaozhanqi

In response to this:

Bug 1992507: Use prometheus rule annotations comply with the OpenShift alerting guidelines

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[creydr]

/retest

[creydr]

/retest

[creydr]

/retest

[creydr]

/retest

[creydr]

/retest

[creydr]

/retest

[abhat]

/lgtm
/approve

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abhat, creydr

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [abhat]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-bot]

/retest-required

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest-required

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest-required

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest-required

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest-required

Please review the full test history for this PR and help us cut down flakes.

[openshift-ci]

@creydr: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Rerun command
ci/prow/e2e-gcp-ovn-upgrade	20d92b3	link	/test e2e-gcp-ovn-upgrade
ci/prow/e2e-openstack-ovn	20d92b3	link	/test e2e-openstack-ovn

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[openshift-bot]

/retest-required

Please review the full test history for this PR and help us cut down flakes.

[openshift-ci]

@creydr: All pull requests linked via external trackers have merged:

• openshift/cluster-network-operator#1181

Bugzilla bug 1992507 has been moved to the MODIFIED state.

In response to this:

Bug 1992507: Use prometheus rule annotations comply with the OpenShift alerting guidelines

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.