feat: add backup metrics to grafana dashboard #10521
Conversation
lenaschoenburg
commented
Sep 27, 2022
•
lenaschoenburg
commented
monitor/grafana/zeebe.json
Outdated
| "targets": [ | ||
| { | ||
| "exemplar": false, | ||
| "expr": "min(max_over_time(zeebe_backup_operations_in_progress{namespace=~\"$namespace\", partition=~\"$partition\", pod=~\"$pod\", operation=\"take\"}[5m])) by (partition)", |
There was a problem hiding this comment.
❓ This query is a bit weird. I tried to remove time series where a gauge would be stuck at a specific value. This happened during testing when a broker would transition to follower while taking a backup. Because that follower never saw the backup complete or fail it would still report an in-progress backup.
I've tried to explain the query I came up here but ultimately I'm not sure why exactly this works or even if it is correct at all 🙈
There was a problem hiding this comment.
May be we should also update the in progress metrics while closing BackupService. If the node restarts, the metrics is reset anyway, I think, as observed in other metrics. Can you verify it? If it works, then we probably don't need this complex query.
In this query, it is not clear what would be the value if inprogress backup takes more than 5m. Will max_over_time query ignored it and shows the result as 0?
There was a problem hiding this comment.
May be we should also update the in progress metrics while closing BackupService
Good idea. Alternatively we could update when marking the in-progress backup as failed during startup of a new BackupService.
There was a problem hiding this comment.
I fixed it by resetting the in-progress counters to 0 when the BackupService actor closes. Query is adjusted too.
Test Results 763 files - 169 763 suites - 169 1h 50m 26s ⏱️ - 14m 33s For more details on these failures, see this check. Results for commit 388cf10. ± Comparison against base commit 751ac05. ♻️ This comment has been updated with latest results. |
|
@oleschoenburg Could you also share the final dashboard view here? Just for reference. |
monitor/grafana/zeebe.json
Outdated
| "targets": [ | ||
| { | ||
| "exemplar": false, | ||
| "expr": "min(max_over_time(zeebe_backup_operations_in_progress{namespace=~\"$namespace\", partition=~\"$partition\", pod=~\"$pod\", operation=\"take\"}[5m])) by (partition)", |
There was a problem hiding this comment.
May be we should also update the in progress metrics while closing BackupService. If the node restarts, the metrics is reset anyway, I think, as observed in other metrics. Can you verify it? If it works, then we probably don't need this complex query.
In this query, it is not clear what would be the value if inprogress backup takes more than 5m. Will max_over_time query ignored it and shows the result as 0?
lenaschoenburg
force-pushed
the
os-backup-metrics
branch
from
e0db00f
to
0edd599
Compare
lenaschoenburg
force-pushed
the
os-backup-metrics
branch
from
0edd599
to
4a9af3c
Compare
lenaschoenburg
force-pushed
the
os-backup-metrics
branch
from
4a9af3c
to
388cf10
Compare
lenaschoenburg
force-pushed
the
os-backup-metrics
branch
from
388cf10
to
ebe5a1b
Compare
|
I added the correct unit for backup latency, thanks for spotting this 👍 bors r+ |
|
Build succeeded: |
|
Successfully created backport PR #10544 for |
Zelldon
added
the
version:8.1.0
