TRT-1357: Publish disruption delta metrics #1330
Conversation
dgoodwin
changed the title
|
@dgoodwin: This pull request references TRT-1357 which is a valid jira issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
openshift-ci-robot
added
the
jira/valid-reference
openshift-ci
bot
added
the
approved
dgoodwin
force-pushed
the
disruption-metrics
branch
2 times, most recently
from
32facb7
to
1eeb58a
Compare
pkg/sippyserver/metrics/metrics.go
Outdated
| /* TODO: don't commit this | ||
|
|
||
| if err := refreshComponentReadinessMetrics(bqc); err != nil { | ||
| log.WithError(err).Error("error refreshing component readiness metrics") | ||
| } | ||
|
|
||
| */ |
There was a problem hiding this comment.
Looks like you committed this :)
There was a problem hiding this comment.
I know myself. :) Thanks!
dgoodwin
force-pushed
the
disruption-metrics
branch
from
1eeb58a
to
6306ee8
Compare
pkg/api/disruption_report.go
Outdated
| generator := disruptionReportGenerator{ | ||
| client: client.BQ, | ||
| viewName: "BackendDisruptionPercentilesDeltaCurrentVsPrevGA", | ||
| } | ||
|
|
||
| return getReportFromCacheOrGenerate[apitype.DisruptionReport](client.Cache, generator, generator.GenerateReport, apitype.DisruptionReport{}) |
There was a problem hiding this comment.
I don't think the caching will work, getReportFromCacheOrGenerate will use the JSON marshalled generator as the cache key, but there's no public fields in generator so it'll probably just marshal to {} (getReportFromCacheORGenerate should probably check for that).
Make viewName public (ViewName)
There was a problem hiding this comment.
Is just making the ViewName public enough there?
There was a problem hiding this comment.
Yea, it'll make the cache key {"ViewName": "BackEnd....."} instead of {}
There was a problem hiding this comment.
Ok added, and the cache will error if it sees an empty json struct
dgoodwin
force-pushed
the
disruption-metrics
branch
from
6306ee8
to
aee1f5c
Compare
pkg/api/component_report.go
Outdated
| if bytes.Equal(cacheKey, []byte("{}")) { | ||
| return defaultVal, []error{fmt.Errorf("cache key serialized to an empty json struct, must have public fields")} | ||
| } |
There was a problem hiding this comment.
I'm not sure this would get noticed, what do you think about panicking like this? #1333
There was a problem hiding this comment.
Updated to remove my version.
Query new views in BigQuery which contain the complex logic, whatever is in those views, we publish a metric for. One of them will need to be updated as part of the GA date tasks. Metrics published for the P50/75/95 delta now compared to both 30 days prior to previous GA, and two weeks prior to two weeks ago. Also includes the delta in PercentageAboveZero, which looks to volatile to alert on now but we'll see if we can get a handle on it. We'll alert on these to notify ourselves of regressions in either.
dgoodwin
force-pushed
the
disruption-metrics
branch
from
aee1f5c
to
b3ac887
Compare
|
@dgoodwin: all tests passed! Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: dgoodwin, stbenjam The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Query new views in BigQuery which contain the complex logic, whatever is
in those views, we publish a metric for. One of them will need to be
updated as part of the GA date tasks.
Metrics published for the P50/75/95 delta now compared to both 30 days
prior to previous GA, and two weeks prior to two weeks ago.
We'll alert on these to notify ourselves of regressions in either.