Fix prometheus rules and improve/fix the SOP #151

camilamacedo86 · 2019-07-12T10:24:29Z

Motivation

https://issues.jboss.org/browse/INTLY-2709 ( raised after discussions over UPS SOP and Prometheus rules and the review over it)

What

Remove the CLI steps from the SOP's
Small fixes in the steps
Fix the MobileSecurityServicePodCount ( it should alert when the pods are < 3 and NOT != of 3 )
Fix the MobileSecurityServiceDown ( it was not working at all and shown the alert when the MSS pod was up and running )
Add steps to scale the MSS pods in the case of some issues regards performance be faced.
Add further information in order to attend the OPS suggestions/requirements

Verification Steps

Check the Sops MMS and Oper
Check here the Prometheus Rule changed: https://prometheus-route-middleware-monitoring.apps.london-4bc4.openshiftworkshop.com/alerts

Checklist:

Code has been tested locally by PR requester
Changes have been successfully verified by another team member

Progress

Finished task
TODO

Additional Notes

camilamacedo86 · 2019-07-12T12:33:41Z

deploy/monitor/mss_prometheus_rule.yaml

@@ -15,7 +15,7 @@ spec:
    - name: general.rules
      rules:
      - alert: MobileSecurityServiceDown
-        expr: absent(up{job="mobile-security-service-application"} == 1)
+        expr: absent(kube_pod_container_status_running{namespace="mobile-security-service",container="application"}==1)
        for: 5m


It was not working here because of this was changed.

should this be >= 1 ?

@camilamacedo86 what about this comment?

It is true. Really tks to get it.

Done it + add further information in the SOPs as was requested/suggested by OPS.

wei-lee · 2019-07-15T08:55:49Z

deploy/monitor/prometheus_rule.yaml

          summary: Pod count for namespace mobile-security-service is {{ printf "%.0f" $value }}. Expected 3 pods. For more information see on the MSS operator https://github.com/aerogear/mobile-security-service-operator"
          sop_url: "https://github.com/aerogear/mobile-security-service-operator/blob/0.2.0/SOP/SOP-operator.adoc"
        expr: |
-          (1-absent(kube_pod_status_ready{condition="true", namespace="mobile-security-service"})) or sum(kube_pod_status_ready{condition="true", namespace="mobile-security-service"}) != 3
+          (1-absent(kube_pod_status_ready{condition="true", namespace="mobile-security-service"})) or sum(kube_pod_status_ready{condition="true", namespace="mobile-security-service"}) < 3


rather than hard coding the namespace value here, is it possible to get the namespace name dynamically? The problem is that the namespace name could be different on different clusters.

cc @grdryn @laurafitzgerald are we using hardcoded namespace in UPS too?

I understand that it is done in the integr8ly ansible scripts as it was done here, for example : https://github.com/integr8ly/installation/pull/687/files#diff-eaa212d002e12d63341d8140d3803235R18. Here it works for the impl of this repo then in the RHMI, it is implemented by the ansible scripts.

Please, let me know if I did not get as you did it, folks. @laurafitzgerald @austincunningham

Ideally for the monitoring resources for the service (rather than the operator itself (although maybe it could provision for itself also, like it provisions its own Service for metrics exposure), the operator would be provisioning those (after checking that the relevant CRDs exist on the cluster already). I think that's the best eventual solution (maybe not necessary to do for this release, and if you're correct about the integreatly installer, then that's probably enough for now).

The keylock is not managed and doing the things in the way that you are proposing here as well. However, in the future maybe it could be improved horizontally in all projects as you suggested. For now, IMHO shows fine move within a way that will work for the current repo and will allow us to use it to create easily the templates to manage it by ansible scripts as it shows been doing so far.

I understand that we agreed to move forward with this approach so far internally. Please, let me know if has anything else that we should change here.

let's create some issues to capture future improvements.

wei-lee · 2019-07-15T11:18:37Z

deploy/monitor/prometheus_rule.yaml

        annotations:
-          description: The Pod count for the mobile-security-service has changed in the last 5 minutes.
+          description: The Pod count for the mobile-security-service has changed in the last 5 minutes and has not the minimal quantity required.


The Pod count for the mobile-security-service has changed in the last 5 minutes and is less than the minimal required value.

Done 👍 . However, after any change in the PR files, it is losing the approval. Please, could you do it again for we are able to merge it?

wei-lee

a small suggestion to update the description. otherwise LGTM.

camilamacedo86 · 2019-07-15T11:24:58Z

Hi @maleck13 @Nanyte25 please feel free to make suggestions for any part of the SOP's files.
See here: https://github.com/aerogear/mobile-security-service-operator/tree/master/SOP.

…stions/requirements

camilamacedo86 · 2019-07-15T16:41:55Z

Hi @wei-lee I think we are able to move forward here now.

camilamacedo86 force-pushed the fix-prometheus-sop branch from 05497c0 to f8c2613 Compare July 12, 2019 10:26

fix(Prometheus) : Fix MobileSecurityServiceDown

898304f

camilamacedo86 force-pushed the fix-prometheus-sop branch 2 times, most recently from 201cdd0 to 9109769 Compare July 12, 2019 11:11

camilamacedo86 requested review from austincunningham, aliok, laurafitzgerald, wei-lee and davidffrench July 12, 2019 11:15

camilamacedo86 force-pushed the fix-prometheus-sop branch from 9109769 to dd40c3b Compare July 12, 2019 11:18

camilamacedo86 requested a review from grdryn July 12, 2019 11:19

camilamacedo86 changed the title ~~fix(SOP and Prometheus rules): explain how to increase the podsin the…~~ Fix prometheus rules and improve/fix the SOP Jul 12, 2019

camilamacedo86 force-pushed the fix-prometheus-sop branch 3 times, most recently from 827908a to 4a57e61 Compare July 12, 2019 12:28

camilamacedo86 commented Jul 12, 2019

View reviewed changes

camilamacedo86 force-pushed the fix-prometheus-sop branch from 4a57e61 to aaa4ac3 Compare July 12, 2019 12:35

wei-lee reviewed Jul 15, 2019

View reviewed changes

camilamacedo86 requested a review from wei-lee July 15, 2019 10:37

wei-lee reviewed Jul 15, 2019

View reviewed changes

wei-lee previously approved these changes Jul 15, 2019

View reviewed changes

camilamacedo86 requested review from maleck13 and Nanyte25 July 15, 2019 11:23

camilamacedo86 dismissed wei-lee’s stale review via 1266e09 July 15, 2019 11:28

camilamacedo86 force-pushed the fix-prometheus-sop branch from aaa4ac3 to 1266e09 Compare July 15, 2019 11:28

camilamacedo86 requested a review from wei-lee July 15, 2019 11:29

camilamacedo86 added 3 commits July 15, 2019 09:23

fix(Prometheus) : Fix MobileSecurityServicePodcount

fdb3a15

fix(SOP) : Improve and fix sop in order to add steps to scale the pod

7bb2259

fix(SOP/Prometheus) : Changeog update

3deb906

fix(operator-sop): fix steps

2c87770

camilamacedo86 force-pushed the fix-prometheus-sop branch from 1266e09 to 2c87770 Compare July 15, 2019 12:23

feat(SOPs) : Add further information in order to attend the OPS sugge…

3729f93

…stions/requirements

camilamacedo86 force-pushed the fix-prometheus-sop branch from 551ddf6 to 3729f93 Compare July 15, 2019 12:49

camilamacedo86 requested a review from pat-cremin July 15, 2019 16:41

wei-lee approved these changes Jul 16, 2019

View reviewed changes

camilamacedo86 merged commit 82717bd into aerogear:master Jul 16, 2019

camilamacedo86 deleted the fix-prometheus-sop branch July 16, 2019 08:35

camilamacedo86 mentioned this pull request Jul 16, 2019

fix(prometheus) : fix review suggestion which causes mal format issues #152

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix prometheus rules and improve/fix the SOP #151

Fix prometheus rules and improve/fix the SOP #151

camilamacedo86 commented Jul 12, 2019 •

edited

Loading

camilamacedo86 Jul 12, 2019 •

edited

Loading

wei-lee Jul 15, 2019

wei-lee Jul 15, 2019

camilamacedo86 Jul 15, 2019

camilamacedo86 Jul 15, 2019

wei-lee Jul 15, 2019

camilamacedo86 Jul 15, 2019 •

edited

Loading

grdryn Jul 15, 2019

camilamacedo86 Jul 15, 2019 •

edited

Loading

camilamacedo86 Jul 15, 2019 •

edited

Loading

wei-lee Jul 15, 2019

wei-lee Jul 15, 2019

camilamacedo86 Jul 15, 2019

wei-lee left a comment

camilamacedo86 commented Jul 15, 2019 •

edited

Loading

camilamacedo86 commented Jul 15, 2019

Fix prometheus rules and improve/fix the SOP #151

Fix prometheus rules and improve/fix the SOP #151

Conversation

camilamacedo86 commented Jul 12, 2019 • edited Loading

Motivation

What

Verification Steps

Checklist:

Progress

Additional Notes

camilamacedo86 Jul 12, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

camilamacedo86 Jul 15, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

camilamacedo86 Jul 15, 2019 • edited Loading

Choose a reason for hiding this comment

camilamacedo86 Jul 15, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wei-lee left a comment

Choose a reason for hiding this comment

camilamacedo86 commented Jul 15, 2019 • edited Loading

camilamacedo86 commented Jul 15, 2019

camilamacedo86 commented Jul 12, 2019 •

edited

Loading

camilamacedo86 Jul 12, 2019 •

edited

Loading

camilamacedo86 Jul 15, 2019 •

edited

Loading

camilamacedo86 Jul 15, 2019 •

edited

Loading

camilamacedo86 Jul 15, 2019 •

edited

Loading

camilamacedo86 commented Jul 15, 2019 •

edited

Loading