OCPBUGS-22422: Fix PodStartupStorageOperationsFailing alert

[jsafrane]

The alert fired only when there was a successful operation on a node. If a CSI driver / volume plugin always failed on a node, the alert was silent, because increase(storage_operation_duration_seconds_count{status = "success")[5m]) has no data.

Trying to fix it with more elaborate promql. When increase(...{status = "success"} has no data, supplement its value by increase(...{status != "success"}) * 0. We know that non-successful metric exists, so multiply it by zero to get a metric of successes with zero values.

Then use sum without(status) and ignoring(status) to merge the metrics by all other labels than status.

cc @openshift/storage

[openshift-ci-robot]

@jsafrane: This pull request references Jira Issue OCPBUGS-22422, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.16.0) matches configured target version for branch (4.16.0)
• bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @ropatil010

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

The alert fired only when there was a successful operation on a node. If a CSI driver / volume plugin always failed on a node, the alert was silent, because increase(storage_operation_duration_seconds_count{status = "success")[5m]) has no data.

Trying to fix it with more elaborate promql. When increase(...{status = "success"} has no data, supplement its value by increase(...{status != "success"}) * 0. We know that non-successful metric exists, so multiply it by zero to get a metric of successes with zero values.

Then use sum without(status) and ignoring(status) to merge the metrics by all other labels than status.

cc @openshift/storage

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jsafrane

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [jsafrane]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jsafrane

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [jsafrane]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jsafrane]

/retest-required

[Phaow]

/retest-required

[RomanBednar]

/lgtm

[openshift-ci-robot]

/retest-required

Remaining retests: 0 against base HEAD e092fb8 and 2 for PR HEAD 082a634 in total

[openshift-bot]

/jira refresh

The requirements for Jira bugs have changed (Jira issues linked to PRs on main branch need to target different OCP), recalculating validity.

[openshift-ci-robot]

@openshift-bot: This pull request references Jira Issue OCPBUGS-22422, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.16.0) matches configured target version for branch (4.16.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @ropatil010

In response to this:

/jira refresh

The requirements for Jira bugs have changed (Jira issues linked to PRs on main branch need to target different OCP), recalculating validity.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[jsafrane]

/retest-required

[openshift-ci]

@jsafrane: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-ibmcloud-csi	082a634	link	false	/test e2e-ibmcloud-csi

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[dobsonj]

/retest-required

[openshift-ci-robot]

@jsafrane: Jira Issue OCPBUGS-22422: All pull requests linked via external trackers have merged:

• openshift/cluster-storage-operator#436

Jira Issue OCPBUGS-22422 has been moved to the MODIFIED state.

In response to this:

The alert fired only when there was a successful operation on a node. If a CSI driver / volume plugin always failed on a node, the alert was silent, because increase(storage_operation_duration_seconds_count{status = "success")[5m]) has no data.

Trying to fix it with more elaborate promql. When increase(...{status = "success"} has no data, supplement its value by increase(...{status != "success"}) * 0. We know that non-successful metric exists, so multiply it by zero to get a metric of successes with zero values.

Then use sum without(status) and ignoring(status) to merge the metrics by all other labels than status.

cc @openshift/storage

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-bot]

[ART PR BUILD NOTIFIER]

This PR has been included in build ose-cluster-storage-operator-container-v4.16.0-202401032310.p0.g466d005.assembly.stream for distgit cluster-storage-operator.
All builds following this will include this PR.