Added 'Mimir / Overview resources' dashboard #3481

pracucci · 2022-11-21T16:36:52Z

What this PR does

In this PR I'm proposing a new dashboard called "Mimir / Overview resources" which shows an overview over a Mimir cluster resources utilization (CPU, memory, disk). The Mimir components are grouped into "Writes", "Reads" and "Backend" and the dashboard both work with microservices and read-write deployment mode.

Note to reviewers:

I left few GitHub comments to explain some of the diff / changes
If you think reviewing this PR is too difficult, I can open a preliminary PR with the utilities refactoring

Example of microservices deployment:

Example of read-write deployment:

Which issue(s) this PR fixes or relates to

Part of #3361

Checklist

Tests updated
Documentation added
CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

pracucci · 2022-11-21T16:38:48Z

operations/mimir-mixin-compiled/dashboards/mimir-alertmanager-resources.json

@@ -526,7 +526,7 @@
                  "steppedLine": false,
                  "targets": [
                     {
-                        "expr": "sum by(instance, pod, device) (\n  rate(\n    node_disk_written_bytes_total[$__rate_interval]\n  )\n)\n+\nignoring(pod) group_right() (\n  label_replace(\n    count by(\n      instance,\n      pod,\n      device\n    )\n    (\n      container_fs_writes_bytes_total{\n        cluster=~\"$cluster\", namespace=~\"$namespace\",\n        container=\"alertmanager\",\n        device!~\".*sda.*\"\n      }\n    ),\n    \"device\",\n    \"$1\",\n    \"device\",\n    \"/dev/(.*)\"\n  ) * 0\n)\n\n",
+                        "expr": "sum by(instance, pod, device) (\n  rate(\n    node_disk_written_bytes_total[$__rate_interval]\n  )\n)\n+\nignoring(pod) group_right() (\n  label_replace(\n    count by(\n      instance,\n      pod,\n      device\n    )\n    (\n      container_fs_writes_bytes_total{\n        cluster=~\"$cluster\", namespace=~\"$namespace\",\n        container=~\"alertmanager\",\n        device!~\".*sda.*\"\n      }\n    ),\n    \"device\",\n    \"$1\",\n    \"device\",\n    \"/dev/(.*)\"\n  ) * 0\n)\n\n",


Note to reviewers: few other dashboards have this change to disk I/O query because I've changed the container="..." matcher to a regex matcher container=~"..."

pracucci · 2022-11-21T16:39:36Z

operations/mimir-mixin-compiled/dashboards/mimir-alertmanager-resources.json

@@ -690,7 +690,7 @@
                  "steppedLine": false,
                  "targets": [
                     {
-                        "expr": "max by(persistentvolumeclaim) (\n  kubelet_volume_stats_used_bytes{cluster=~\"$cluster\", namespace=~\"$namespace\"} /\n  kubelet_volume_stats_capacity_bytes{cluster=~\"$cluster\", namespace=~\"$namespace\"}\n)\nand\ncount by(persistentvolumeclaim) (\n  kube_persistentvolumeclaim_labels{\n    cluster=~\"$cluster\", namespace=~\"$namespace\",\n    label_name=\"alertmanager\"\n  }\n)\n",
+                        "expr": "max by(persistentvolumeclaim) (\n  kubelet_volume_stats_used_bytes{cluster=~\"$cluster\", namespace=~\"$namespace\"} /\n  kubelet_volume_stats_capacity_bytes{cluster=~\"$cluster\", namespace=~\"$namespace\"}\n)\nand\ncount by(persistentvolumeclaim) (\n  kube_persistentvolumeclaim_labels{\n    cluster=~\"$cluster\", namespace=~\"$namespace\",\n    label_name=~\"(alertmanager).*\"\n  }\n)\n",


Note to reviewers: to simplify the multi-zone matching, I've applied the .* suffix to all label_name matchers.

pracucci · 2022-11-21T16:42:51Z

operations/mimir-mixin/dashboards/dashboard-utils.libsonnet


-  resourcesPanelQueries(metric, instanceName)::
+  resourceUtilizationQuery(metric, instanceName)::


Note to reviewers: introduced a new utility function to generate the resource utilization query (without request/limit queries).

Why did you choose to not have requests/limits? I think we found them helpful when they were added on the other resource dashboards

I imagine because those will be different when this dashboard is used on a microservices deployment. In that case we can repeat the panel for every different container we have.

Alternatively, we can display the memory utilization as a % of the limit instead of an absolute value. I think I prefer this option.

I imagine because those will be different when this dashboard is used on a microservices deployment.

For this reason.

In that case we can repeat the panel for every different container we have.

Well, that goes against the idea of having an "Overview". If you want to go deep, you open the "Writes resources" or "Reads resources" dashboard.

pracucci · 2022-11-21T16:43:24Z

operations/mimir-mixin/dashboards/dashboard-utils.libsonnet

      $._config.resources_panel_queries[$._config.deployment_type]['%s_usage' % metric] % {
        instance: $._config.per_instance_label,
        namespace: $.namespaceMatcher(),
        instanceName: instanceName,
      },
+
+  resourceUtilizationAndLimitQueries(metric, instanceName)::


Note to reviewers: I renamed resourcesPanelQueries() to resourceUtilizationAndLimitQueries() to clarify it also include resources request/limit.

johannaratliff

LGTM

dimitarvdimitrov · 2022-11-22T08:20:50Z

operations/mimir-mixin/dashboards/dashboard-utils.libsonnet


-  resourcesPanelQueries(metric, instanceName)::
+  resourceUtilizationQuery(metric, instanceName)::


Why did you choose to not have requests/limits? I think we found them helpful when they were added on the other resource dashboards

I imagine because those will be different when this dashboard is used on a microservices deployment. In that case we can repeat the panel for every different container we have.

Alternatively, we can display the memory utilization as a % of the limit instead of an absolute value. I think I prefer this option.

operations/mimir-mixin/dashboards/dashboard-utils.libsonnet

operations/mimir-mixin/dashboards/overview-resources.libsonnet

Signed-off-by: Marco Pracucci <marco@pracucci.com>

…oards Signed-off-by: Marco Pracucci <marco@pracucci.com>

dimitarvdimitrov

LGTM, thanks for addressing my feedback

osg-grafana · 2022-11-22T09:55:57Z

operations/mimir-mixin-compiled/dashboards/mimir-overview.json

@@ -163,7 +163,7 @@
            "height": "250px",
            "panels": [
               {
-                  "content": "These panels show an overview on the write path. \nVisit the following specific dashboards to drill down into the write path:\n\n- <a target=\"_blank\" href=\"./d/8280707b8f16e7b87b840fc1cc92d4c5/mimir-writes?${__url_time_range}&${__all_variables}\">Writes</a>\n- <a target=\"_blank\" href=\"./d/bc9160e50b52e89e0e49c840fea3d379/mimir-writes-resources?${__url_time_range}&${__all_variables}\">Writes resources</a>\n- <a target=\"_blank\" href=\"./d/978c1cb452585c96697a238eaac7fe2d/mimir-writes-networking?${__url_time_range}&${__all_variables}\">Writes networking</a>\n",
+                  "content": "These panels show an overview on the write path. \nVisit the following specific dashboards to drill down into the write path:\n\n- <a target=\"_blank\" href=\"./d/8280707b8f16e7b87b840fc1cc92d4c5/mimir-writes?${__url_time_range}&${__all_variables}\">Writes</a>\n- <a target=\"_blank\" href=\"./d/bc9160e50b52e89e0e49c840fea3d379/mimir-writes-resources?${__url_time_range}&${__all_variables}\">Writes resources</a>\n- <a target=\"_blank\" href=\"./d/978c1cb452585c96697a238eaac7fe2d/mimir-writes-networking?${__url_time_range}&${__all_variables}\">Writes networking</a>\n- <a target=\"_blank\" href=\"./d/a9b92d3c4d1af325d872a9e9a7083d71/mimir-overview-resources?${__url_time_range}&${__all_variables}\">Overview resources</a>\n",


This is problematic because "writes" is a verb and a noun in this context. Is it "Writes’ resources"? or is it "[Something] writes resources"?

My impression is that this is multiple writes’ resources rather than a single write’s resources.

Please simplify the multiple prepositions (down into) in "Visit the following specific dashboards to drill down into the write path:" Suggestion: To examine the write path in detail, see a specific dashboard:

* Added 'Mimir / Overview resources' dashboard Signed-off-by: Marco Pracucci <marco@pracucci.com> * Simplified resourceUtilizationQuery() Signed-off-by: Marco Pracucci <marco@pracucci.com> * Do not allow to select multiple namespaces like other resources dashboards Signed-off-by: Marco Pracucci <marco@pracucci.com> Signed-off-by: Marco Pracucci <marco@pracucci.com>

pracucci commented Nov 21, 2022

View reviewed changes

pracucci force-pushed the add-overview-resources-dashboard branch from ad54cdf to ca43ec6 Compare November 21, 2022 16:56

pracucci marked this pull request as ready for review November 21, 2022 16:57

pracucci requested review from a team as code owners November 21, 2022 16:57

johannaratliff approved these changes Nov 21, 2022

View reviewed changes

dimitarvdimitrov reviewed Nov 22, 2022

View reviewed changes

pracucci added 2 commits November 22, 2022 09:42

Added 'Mimir / Overview resources' dashboard

acc53be

Signed-off-by: Marco Pracucci <marco@pracucci.com>

Simplified resourceUtilizationQuery()

f8c95c2

Signed-off-by: Marco Pracucci <marco@pracucci.com>

pracucci force-pushed the add-overview-resources-dashboard branch from ca43ec6 to f8c95c2 Compare November 22, 2022 08:45

Do not allow to select multiple namespaces like other resources dashb…

6e52309

…oards Signed-off-by: Marco Pracucci <marco@pracucci.com>

dimitarvdimitrov approved these changes Nov 22, 2022

View reviewed changes

pracucci merged commit 4e7a423 into main Nov 22, 2022

pracucci deleted the add-overview-resources-dashboard branch November 22, 2022 09:18

osg-grafana reviewed Nov 22, 2022

View reviewed changes

osg-grafana added the type/docs Improvements or additions to documentation label Nov 22, 2022

This was referenced Nov 22, 2022

Added 'Mimir / Overview networking' dashboard #3487

Merged

Improve phrasing in Overview dashboard #3488

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added 'Mimir / Overview resources' dashboard #3481

Added 'Mimir / Overview resources' dashboard #3481

pracucci commented Nov 21, 2022 •

edited

Loading

pracucci Nov 21, 2022

pracucci Nov 21, 2022

pracucci Nov 21, 2022

dimitarvdimitrov Nov 22, 2022

pracucci Nov 22, 2022 •

edited

Loading

pracucci Nov 21, 2022

johannaratliff left a comment

dimitarvdimitrov Nov 22, 2022

dimitarvdimitrov left a comment

osg-grafana Nov 22, 2022

osg-grafana Nov 22, 2022

osg-grafana Nov 22, 2022

pracucci Nov 22, 2022


		resourcesPanelQueries(metric, instanceName)::
		resourceUtilizationQuery(metric, instanceName)::

Added 'Mimir / Overview resources' dashboard #3481

Added 'Mimir / Overview resources' dashboard #3481

Conversation

pracucci commented Nov 21, 2022 • edited Loading

What this PR does

Which issue(s) this PR fixes or relates to

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pracucci Nov 22, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

johannaratliff left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dimitarvdimitrov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pracucci commented Nov 21, 2022 •

edited

Loading

pracucci Nov 22, 2022 •

edited

Loading