Prepare insight metrics structure for adding service_name label #4227
Conversation
…s cache with respect to service_name
Ferril
added
the
pr:no public docs
Ferril
changed the title
| all_response_time_seconds = [int(response_time.total_seconds()) for response_time in all_response_time] | ||
|
|
||
| metric_alert_group_response_time[integration.id] = { | ||
| # count alert groups with `service_name` label group by label value |
There was a problem hiding this comment.
Have we tried these queries on production db? Do we need to apply any optimizations?
There was a problem hiding this comment.
Yes, the first call for response time is slow (>20s), subsequent calls work faster (~2s). I'm checking how to optimize this.
UPD: to speed up queries with service_name label I added filter by organization, what makes them to use label table composite index
There was a problem hiding this comment.
+1 to optimize as possible (using the index improved things?). In any case, this shouldn't affect the response time in the scrape endpoint right? (there we are still getting the data from cache and returning it, correct?)
There was a problem hiding this comment.
+1 to optimize as possible (using the index improved things?)
the slowest call from my tests is ~10s now. It uses readonly db, so I don't think this would be a problem 🤔
n any case, this shouldn't affect the response time in the scrape endpoint right? (there we are still getting the data from cache and returning it, correct?)
Correct, we get all data from the cache and don't do any calculations there
| @@ -61,3 +66,6 @@ class RecalculateOrgMetricsDict(typing.TypedDict): | |||
|
|
|||
| METRICS_ORGANIZATIONS_IDS = "metrics_organizations_ids" | |||
| METRICS_ORGANIZATIONS_IDS_CACHE_TIMEOUT = 3600 # 1 hour | |||
|
|
|||
| SERVICE_LABEL = "service_name" | |||
| NO_SERVICE_VALUE = "No service" | |||
There was a problem hiding this comment.
Just curious, have we considered "Unnamed service" or "No name service" instead? ("No service" sounds as something broken? :-))
There was a problem hiding this comment.
It's like "No team", but for services :) And SLO uses "No service" as well, so it looks consistent
| all_response_time_seconds = [int(response_time.total_seconds()) for response_time in all_response_time] | ||
|
|
||
| metric_alert_group_response_time[integration.id] = { | ||
| # count alert groups with `service_name` label group by label value |
There was a problem hiding this comment.
+1 to optimize as possible (using the index improved things?). In any case, this shouldn't affect the response time in the scrape endpoint right? (there we are still getting the data from cache and returning it, correct?)
github-merge-queue
bot
removed this pull request from the merge queue due to failed status checks
github-merge-queue
bot
removed this pull request from the merge queue due to failed status checks
# What this PR does Adds `service_name` label to insight metrics NOTE: It is related to [this PR](#4227) and should be merged no sooner than two days after the next release (current release version is 1.4.4), because we need to wait for the metrics cache to be updated for all organizations (uses the new cache structure with `services`) ## Which issue(s) this PR closes Related to grafana/oncall-private#2610 ## Checklist - [x] Unit, integration, and e2e (if applicable) tests updated - [x] Documentation added (or `pr:no public docs` PR label added if not required) - [x] Added the relevant release notes label (see labels prefixed w/ `release:`). These labels dictate how your PR will show up in the autogenerated release notes.
What this PR does
Prepare insight metrics for adding
service_namelabel.This PR updates metrics cache structure, supporting both old and new version of cache.
service_namelabel can be added with additional PR when all metric cache is updated.Which issue(s) this PR closes
https://github.com/grafana/oncall-private/issues/2610
Checklist
pr:no public docsPR label added if not required)release:). These labels dictate how your PR willshow up in the autogenerated release notes.