Bug 2014391: ceph: fixing the queries for alerts 'CephMgrIsAbsent' and 'CephMgrIsM… #301
Conversation
…issingReplicas'
CephMgrIsAbsent
----------------
This alert initially had the following query
absent(up{job="rook-ceph-mgr"})
which will fire when the 'up' query is not present, but had two flows
a. it will not be fired if 'up' provides a result with ZERO value
b. it will not give any fields in the metric, so 'namespace' was missing
when the above query was replaced with the following,
up{job="rook-ceph-mgr"} == 0
query had the following shortage
a. whenever mgr pod is completely down (like 'replicas' set to ZERO
and 'mgr' is not coming up), 'up' query will not give any result.
Thus we had to combine both the queries to get results in both the scenarios.
CephMgrIsMissingReplicas
------------------------
This query previously was,
sum(up{job="rook-ceph-mgr"}) < 1
had the same structure as the above (Absent) query, but it's
intention was to check the no: of 'replicas' count for ceph mgr.
Now it is changed to a kube query which handles the replicas count.
Signed-off-by: Arun Kumar Mohan <amohan@redhat.com>
(cherry picked from commit cfa2c2d)
openshift-ci
bot
added
bugzilla/severity-high
|
@agarwal-mudit: This pull request references Bugzilla bug 2014391, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker. 2 validation(s) were run on this bug
Requesting review from QA contact: In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
@openshift-ci[bot]: GitHub didn't allow me to request PR reviews from the following users: ebenahar. Note that only red-hat-storage members and repo collaborators can review this PR, and authors cannot review their own PRs. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: agarwal-mudit, leseb The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
@agarwal-mudit: All pull requests linked via external trackers have merged: Bugzilla bug 2014391 has been moved to the MODIFIED state. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
…issingReplicas'
CephMgrIsAbsent
This alert initially had the following query
absent(up{job="rook-ceph-mgr"})
which will fire when the 'up' query is not present, but had two flows
a. it will not be fired if 'up' provides a result with ZERO value
b. it will not give any fields in the metric, so 'namespace' was missing
when the above query was replaced with the following,
up{job="rook-ceph-mgr"} == 0
query had the following shortage
a. whenever mgr pod is completely down (like 'replicas' set to ZERO
and 'mgr' is not coming up), 'up' query will not give any result.
Thus we had to combine both the queries to get results in both the scenarios.
CephMgrIsMissingReplicas
This query previously was,
sum(up{job="rook-ceph-mgr"}) < 1
had the same structure as the above (Absent) query, but it's
intention was to check the no: of 'replicas' count for ceph mgr.
Now it is changed to a kube query which handles the replicas count.
Signed-off-by: Arun Kumar Mohan amohan@redhat.com
(cherry picked from commit cfa2c2d)
Description of your changes:
Which issue is resolved by this Pull Request:
Resolves #
Checklist:
make codegen) has been run to update object specifications, if necessary.