OADP- 2300 OADP Metric Collection #62710

CarmiWisemon · 2023-07-23T10:43:18Z

OADP - 1.2.1
OCP - 4.11+

Resolves - https://issues.redhat.com/browse/OADP-2300

Preview - https://62710--docspreview.netlify.app/openshift-enterprise/latest/backup_and_restore/application_backup_and_restore/troubleshooting.html#oadp-monitoring_oadp-troubleshooting

ocpdocs-previewbot · 2023-07-23T10:52:27Z

🤖 Updated build preview is available at:
https://62710--docspreview.netlify.app

Build log: https://circleci.com/gh/ocpdocs-previewbot/openshift-docs/21729

modules/oadp-monitoring.adoc

mpryc

@CarmiWisemon Thanks.

I have reviewed updated docs, looks fine to me.

One question which came to my mind when I was looking for them. They are under:

https://62710--docspreview.netlify.app/openshift-enterprise/latest/backup_and_restore/application_backup_and_restore/troubleshooting.html#oadp-monitoring_oadp-troubleshooting

Should the monitoring be outside of Troubleshooting ? (same level as Troubleshooting, so one level up) or inside "Advanced OADP features and functionalities"

Some monitoring parts are allowing to receive alerts when backup fails, but this isn't logically troubleshooting to me, which should be more related to OADP issues, misconfigurations or problems that affects the OADP itself.

Monitoring can alert when some problems may happen, but metrics also allows to monitor number of backups, it's duration, size, so not really under troubleshooting umbrella.

CarmiWisemon · 2023-07-31T08:25:26Z

@mpryc
You are correct that it may not be in the ideal location at the moment. We are rearranging the OADP documentation and at present it's in a place holder. Once we rearrange the documentation, the Monitoring section will move somewhere else.

Thank you very much for all of your wonderful feedback!

Carmi

modules/oadp-monitoring.adoc

CarmiWisemon · 2023-08-07T11:38:47Z

/label OADP
/label peer-review-needed

adellape

Apologies for the number of comments; some of them are duplicates that I thought worth re-highlighting in-line versus just saying "do this everywhere". Please let me know if you have any questions!

images/oadp-backup-failing-alert.png

modules/oadp-monitoring.adoc

CarmiWisemon · 2023-08-09T15:19:36Z

@adellape
Thank you for your wonderful review, great suggestions and feedback. I made the changes you requested, except for the images that we can't do at this time.
Thanks for your help! Carmi

CarmiWisemon · 2023-08-09T15:20:00Z

/label merge-review-needed

adellape

Would like to hold merge for some of the following remaining issues.

@CarmiWisemon Leaving the merge-review-in-progress label on for now so it stays in my view. Please ping me on Slack or here when it's ready for another look. Thanks!

adellape · 2023-08-09T19:45:08Z

modules/oadp-monitoring-setup.adoc

+$ oc edit configmap cluster-monitoring-config -n openshift-monitoring
+----
+
+.Example output


I guess technically it's not really "output" in this case, since we're showing what we want them to change. Maybe .Example changes just for this block?

Altho I think more guidelines-appropriate would be to break it into 2 actions (and then you don't need the .Example output block title at all):

. Edit the `cluster-monitoring-config` `ConfigMap` object in the `openshift-monitoring` namespace: + [source,terminal] ---- $ oc edit configmap cluster-monitoring-config -n openshift-monitoring ---- . Add or enable the `enableUserWorkload` option in the `data` section's `config.yaml` field: + [source,yaml] ---- apiVersion: v1 data: config.yaml: | enableUserWorkload: true <1> kind: ConfigMap metadata: # ... ---- <1> Add this option or set to `true`

adellape · 2023-08-09T19:47:51Z

modules/oadp-monitoring-setup.adoc

+    enableUserWorkload: true  <1>
+kind: ConfigMap
+metadata:
+# [...]


Suggested change

# [...]

# ...

Per repo guidelines for YAML.

adellape · 2023-08-09T19:48:53Z

modules/oadp-monitoring-setup.adoc

+. Wait a short period of time to verify the User Workload Monitoring Setup by checking if the following components are up and running in the `openshift-user-workload-monitoring` namespace.
+
+


Colon and blank line:

Suggested change

. Wait a short period of time to verify the User Workload Monitoring Setup by checking if the following components are up and running in the `openshift-user-workload-monitoring` namespace.

+

. Wait a short period of time to verify the User Workload Monitoring Setup by checking if the following components are up and running in the `openshift-user-workload-monitoring` namespace:

+

adellape · 2023-08-09T19:49:14Z

modules/oadp-monitoring-setup.adoc

+----
+NAME                                   READY   STATUS    RESTARTS   AGE
+prometheus-operator-6844b4b99c-b57j9   2/2     Running   0          43s
+prometheus-user-workload-0             5/5     Running   0          32s
+prometheus-user-workload-1             5/5     Running   0          32s
+thanos-ruler-user-workload-0           3/3     Running   0          32s
+thanos-ruler-user-workload-1           3/3     Running   0          32s
+----


Missing block title and source:

Suggested change

----

NAME READY STATUS RESTARTS AGE

prometheus-operator-6844b4b99c-b57j9 2/2 Running 0 43s

prometheus-user-workload-0 5/5 Running 0 32s

prometheus-user-workload-1 5/5 Running 0 32s

thanos-ruler-user-workload-0 3/3 Running 0 32s

thanos-ruler-user-workload-1 3/3 Running 0 32s

----

.Example output

[source,terminal]

----

NAME READY STATUS RESTARTS AGE

prometheus-operator-6844b4b99c-b57j9 2/2 Running 0 43s

prometheus-user-workload-0 5/5 Running 0 32s

prometheus-user-workload-1 5/5 Running 0 32s

thanos-ruler-user-workload-0 3/3 Running 0 32s

thanos-ruler-user-workload-1 3/3 Running 0 32s

----

adellape · 2023-08-09T19:50:36Z

modules/oadp-monitoring-setup.adoc

+[source,terminal]
+----
+$ oc get configmap user-workload-monitoring-config -n openshift-user-workload-monitoring
+Error from server (NotFound): configmaps "user-workload-monitoring-config" not found
+
+# We need to create: user-workload-monitoring-config ConfigMap
+----


Still needs to be broken into 2 code blocks:

Suggested change

[source,terminal]

----

$ oc get configmap user-workload-monitoring-config -n openshift-user-workload-monitoring

Error from server (NotFound): configmaps "user-workload-monitoring-config" not found

# We need to create: user-workload-monitoring-config ConfigMap

----

[source,terminal]

----

$ oc get configmap user-workload-monitoring-config -n openshift-user-workload-monitoring

----

+

.Example output

[source,terminal]

----

Error from server (NotFound): configmaps "user-workload-monitoring-config" not found

----

adellape · 2023-08-09T20:47:00Z

modules/oadp-creating-service-monitor.adoc

+
+. Create a `ServiceMonitor` YAML file that matches the existing SVC label, and save the file as `3_create_oadp_service_monitor.yaml`. The service monitor is created in the `openshift-adp` namespace where the `openshift-adp-velero-metrics-svc` service resides.
+
+.Example output


Suggested change

.Example output

.Example `ServiceMonitor` object

adellape · 2023-08-09T20:47:31Z

modules/oadp-creating-service-monitor.adoc

+----
+$ oc apply -f 3_create_oadp_service_monitor.yaml
+servicemonitor.monitoring.coreos.com/oadp-service-monitor created
+----


Break up into 2 blocks, or drop the output.

adellape · 2023-08-09T20:50:35Z

modules/oadp-creating-service-monitor.adoc

+servicemonitor.monitoring.coreos.com/oadp-service-monitor created
+----
+
+. Confirm that the new service monitor is in an *Up* state by using the *Administrator* perspective of the {product-title} web console:


You could also turn this step into a .Verification section, to match the other module:

Suggested change

. Confirm that the new service monitor is in an *Up* state by using the *Administrator* perspective of the {product-title} web console:

.Verification

* Confirm that the new service monitor is in an *Up* state by using the *Administrator* perspective of the {product-title} web console:

^ Either way, since this is also technically a single-step procedure w/ substeps, change . to *. The substeps are ordered here tho, so they can stay ..

adellape · 2023-08-09T20:55:04Z

modules/oadp-list-of-metrics.adoc

+|`kopia_content_get_bytes`
+|Number of bytes retrieved using GetContent
+|Counter
+
+|`kopia_content_get_count`
+|Number of times GetContent() was called
+|Counter
+
+|`kopia_content_get_error_count`
+|Number of times GetContent() was called and the result was an error
+|Counter
+
+|`kopia_content_get_not_found_count`
+|Number of times GetContent() was called and the result was not found
+|Counter
+
+|`kopia_content_write_bytes`
+|Number of bytes passed to WriteContent()
+|Counter
+
+|`kopia_content_write_count`
+|Number of times WriteContent() was called
+|Counter


adellape · 2023-08-09T20:58:20Z

modules/oadp-viewing-metrics-ui.adoc

+* Navigate to the *Observe* -> *Metrics* page:
+** If you are using the *Developer* perspective, follow these steps:
+.. Select *Custom query*, or click on the *Show PromQL* link.
+.. Type the query and click *Enter*.
+** If you are using the *Administrator* perspective, type the expression in the text field and select *Run Queries*.
+
+.OADP metrics query
+
+image::oadp-metrics-query.png[OADP metrics query]


Still suggest moving the screenshot (which shows Developer perspective) up to pair with the Developer step:

Suggested change

* Navigate to the *Observe* -> *Metrics* page:

** If you are using the *Developer* perspective, follow these steps:

.. Select *Custom query*, or click on the *Show PromQL* link.

.. Type the query and click *Enter*.

** If you are using the *Administrator* perspective, type the expression in the text field and select *Run Queries*.

+

.OADP metrics query

image::oadp-metrics-query.png[OADP metrics query]

* Navigate to the *Observe* -> *Metrics* page:

** If you are using the *Developer* perspective, follow these steps:

.. Select *Custom query*, or click on the *Show PromQL* link.

.. Type the query and click *Enter*.

+

.OADP metrics query

image::oadp-metrics-query.png[OADP metrics query]

** If you are using the *Administrator* perspective, type the expression in the text field and select *Run Queries*.

or just remove the + so that it floats after the step is done (and doesn't look attached to the Administrator step anymore):

Suggested change

* Navigate to the *Observe* -> *Metrics* page:

** If you are using the *Developer* perspective, follow these steps:

.. Select *Custom query*, or click on the *Show PromQL* link.

.. Type the query and click *Enter*.

** If you are using the *Administrator* perspective, type the expression in the text field and select *Run Queries*.

+

.OADP metrics query

image::oadp-metrics-query.png[OADP metrics query]

* Navigate to the *Observe* -> *Metrics* page:

** If you are using the *Developer* perspective, follow these steps:

.. Select *Custom query*, or click on the *Show PromQL* link.

.. Type the query and click *Enter*.

** If you are using the *Administrator* perspective, type the expression in the text field and select *Run Queries*.

.OADP metrics query

image::oadp-metrics-query.png[OADP metrics query]

adellape

Thank you for the applied changes. I am going to merge per Slack discussion, however please consider the few remaining items below that could be considered for a quick follow-up PR.

adellape · 2023-08-10T20:56:53Z

modules/oadp-monitoring-setup.adoc

+. Create `user-workload-monitoring-config` ConfigMap for the User Workload Monitoring and save it under `2_configure_user_workload_monitoring.yaml` filename.
+
+.Example output
+[source,yaml]
+
+----
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: user-workload-monitoring-config
+  namespace: openshift-user-workload-monitoring
+data:
+  config.yaml: |
+----
+


This one wasn't addressed.

adellape · 2023-08-10T20:57:24Z

modules/oadp-creating-service-monitor.adoc

+
+.Procedure
+
+. Ensure the `openshift-adp-velero-metrics-svc` exists. It should contain `app.kubernetes.io/name=velero` label which will be used as selector for the `ServiceMonitor` object.


Suggested change

. Ensure the `openshift-adp-velero-metrics-svc` exists. It should contain `app.kubernetes.io/name=velero` label which will be used as selector for the `ServiceMonitor` object.

. Ensure the `openshift-adp-velero-metrics-svc` exists. It should contain `app.kubernetes.io/name=velero` label, which will be used as selector for the `ServiceMonitor` object.

adellape · 2023-08-10T20:58:36Z

modules/oadp-creating-alerting-rule.adoc

+----
+
+.Verification
+. After the Alert is triggered, you can view it in the following ways:


Suggested change

. After the Alert is triggered, you can view it in the following ways:

* After the Alert is triggered, you can view it in the following ways:

adellape · 2023-08-10T21:02:57Z

/cherrypick enterprise-4.14

adellape · 2023-08-10T21:03:00Z

/cherrypick enterprise-4.13

adellape · 2023-08-10T21:03:03Z

/cherrypick enterprise-4.12

adellape · 2023-08-10T21:03:06Z

/cherrypick enterprise-4.11

openshift-cherrypick-robot · 2023-08-10T21:03:44Z

@adellape: new pull request created: #63444

In response to this:

/cherrypick enterprise-4.14

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-cherrypick-robot · 2023-08-10T21:03:49Z

@adellape: new pull request created: #63445

In response to this:

/cherrypick enterprise-4.13

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-cherrypick-robot · 2023-08-10T21:03:53Z

@adellape: new pull request created: #63446

In response to this:

/cherrypick enterprise-4.11

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-cherrypick-robot · 2023-08-10T21:03:59Z

@adellape: new pull request created: #63447

In response to this:

/cherrypick enterprise-4.12

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci bot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Jul 23, 2023

CarmiWisemon force-pushed the oadp2300metric branch 3 times, most recently from 33745d8 to 078924a Compare July 25, 2023 15:04

CarmiWisemon commented Jul 26, 2023

View reviewed changes

modules/oadp-monitoring.adoc Outdated Show resolved Hide resolved

mpryc reviewed Jul 28, 2023

View reviewed changes

modules/oadp-monitoring.adoc Outdated Show resolved Hide resolved

modules/oadp-monitoring.adoc Outdated Show resolved Hide resolved

CarmiWisemon force-pushed the oadp2300metric branch from 078924a to 5443488 Compare July 30, 2023 08:09

mpryc reviewed Jul 31, 2023

View reviewed changes

TzahiAshkenazi reviewed Aug 1, 2023

View reviewed changes

modules/oadp-monitoring.adoc Outdated Show resolved Hide resolved

CarmiWisemon force-pushed the oadp2300metric branch from 57d92dd to 848a8c6 Compare August 3, 2023 08:19

mpryc reviewed Aug 7, 2023

View reviewed changes

modules/oadp-monitoring.adoc Outdated Show resolved Hide resolved

openshift-ci bot added OADP Label for all OADP PRs peer-review-needed Signifies that the peer review team needs to review this PR labels Aug 7, 2023

adellape added the peer-review-in-progress Signifies that the peer review team is reviewing this PR label Aug 7, 2023

adellape self-assigned this Aug 7, 2023

mpryc approved these changes Aug 7, 2023

View reviewed changes

adellape added branch/enterprise-4.11 branch/enterprise-4.12 branch/enterprise-4.13 branch/enterprise-4.14 labels Aug 7, 2023

adellape reviewed Aug 8, 2023

View reviewed changes

adellape removed peer-review-in-progress Signifies that the peer review team is reviewing this PR peer-review-needed Signifies that the peer review team needs to review this PR labels Aug 8, 2023

CarmiWisemon force-pushed the oadp2300metric branch 4 times, most recently from aac547b to 2e42e01 Compare August 9, 2023 14:50

openshift-ci bot added the merge-review-needed Signifies that the merge review team needs to review this PR label Aug 9, 2023

adellape added the merge-review-in-progress Signifies that the merge review team is reviewing this PR label Aug 9, 2023

adellape reviewed Aug 9, 2023

View reviewed changes

CarmiWisemon force-pushed the oadp2300metric branch 2 times, most recently from c75b687 to bf88086 Compare August 10, 2023 09:35

OADP2300 Metrics

c157822

CarmiWisemon force-pushed the oadp2300metric branch from 69cccf3 to c157822 Compare August 10, 2023 11:35

adellape reviewed Aug 10, 2023

View reviewed changes

adellape merged commit ab797d3 into openshift:main Aug 10, 2023
1 check passed

openshift-cherrypick-robot mentioned this pull request Aug 10, 2023

[enterprise-4.14] OADP- 2300 OADP Metric Collection #63444

Merged

openshift-cherrypick-robot mentioned this pull request Aug 10, 2023

[enterprise-4.13] OADP- 2300 OADP Metric Collection #63445

Merged

openshift-cherrypick-robot mentioned this pull request Aug 10, 2023

[enterprise-4.11] OADP- 2300 OADP Metric Collection #63446

Merged

openshift-cherrypick-robot mentioned this pull request Aug 10, 2023

[enterprise-4.12] OADP- 2300 OADP Metric Collection #63447

Merged

adellape removed merge-review-in-progress Signifies that the merge review team is reviewing this PR merge-review-needed Signifies that the merge review team needs to review this PR labels Aug 10, 2023

adellape mentioned this pull request Aug 14, 2023

OADP 2300 Fixes #63514

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OADP- 2300 OADP Metric Collection #62710

OADP- 2300 OADP Metric Collection #62710

CarmiWisemon commented Jul 23, 2023 •

edited

ocpdocs-previewbot commented Jul 23, 2023 •

edited

mpryc left a comment

CarmiWisemon commented Jul 31, 2023

CarmiWisemon commented Aug 7, 2023

adellape left a comment

CarmiWisemon commented Aug 9, 2023

CarmiWisemon commented Aug 9, 2023

adellape left a comment

adellape Aug 9, 2023

adellape Aug 9, 2023

adellape Aug 9, 2023

adellape Aug 9, 2023

adellape Aug 9, 2023

adellape Aug 9, 2023

adellape Aug 9, 2023

adellape Aug 9, 2023

adellape Aug 9, 2023

adellape Aug 9, 2023

adellape left a comment

adellape Aug 10, 2023

adellape Aug 10, 2023

adellape Aug 10, 2023

adellape commented Aug 10, 2023

adellape commented Aug 10, 2023

adellape commented Aug 10, 2023

adellape commented Aug 10, 2023

openshift-cherrypick-robot commented Aug 10, 2023

openshift-cherrypick-robot commented Aug 10, 2023

openshift-cherrypick-robot commented Aug 10, 2023

openshift-cherrypick-robot commented Aug 10, 2023

		. Wait a short period of time to verify the User Workload Monitoring Setup by checking if the following components are up and running in the `openshift-user-workload-monitoring` namespace.
		+

	. Confirm that the new service monitor is in an Up state by using the Administrator perspective of the {product-title} web console:
	.Verification
	* Confirm that the new service monitor is in an Up state by using the Administrator perspective of the {product-title} web console:


		.Procedure

		. Ensure the `openshift-adp-velero-metrics-svc` exists. It should contain `app.kubernetes.io/name=velero` label which will be used as selector for the `ServiceMonitor` object.

	. After the Alert is triggered, you can view it in the following ways:
	* After the Alert is triggered, you can view it in the following ways:

OADP- 2300 OADP Metric Collection #62710

OADP- 2300 OADP Metric Collection #62710

Conversation

CarmiWisemon commented Jul 23, 2023 • edited

ocpdocs-previewbot commented Jul 23, 2023 • edited

mpryc left a comment

Choose a reason for hiding this comment

CarmiWisemon commented Jul 31, 2023

CarmiWisemon commented Aug 7, 2023

adellape left a comment

Choose a reason for hiding this comment

CarmiWisemon commented Aug 9, 2023

CarmiWisemon commented Aug 9, 2023

adellape left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adellape left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adellape commented Aug 10, 2023

adellape commented Aug 10, 2023

adellape commented Aug 10, 2023

adellape commented Aug 10, 2023

openshift-cherrypick-robot commented Aug 10, 2023

openshift-cherrypick-robot commented Aug 10, 2023

openshift-cherrypick-robot commented Aug 10, 2023

openshift-cherrypick-robot commented Aug 10, 2023

CarmiWisemon commented Jul 23, 2023 •

edited

ocpdocs-previewbot commented Jul 23, 2023 •

edited