Add transformation_success_total and transformation_last_status metrics. #70715

immutableT · 2018-11-06T21:56:18Z

Add transformation_success_total and transformation_last_status metrics to apiserver/pkg/storage/value/metrics.go

What type of PR is this?
/kind feature

What this PR does / why we need it:
When managing KMS Encryption feature within a sizable deployment, answering the following monitoring/health questions becomes important:

How many requests is my KMS-Plugin handling?
What is the ratio between successful and failed KMS requests?
Show me all clusters where the last request to KMS-Plugin failed.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

The transformer_failures_total metric is deprecated in favor of transformation_operation_total. The old metric will continue to be populated but will be removed in a future release.

immutableT · 2018-11-06T21:58:47Z

/assign @lavalamp

immutableT · 2018-11-06T21:59:26Z

/assign @liggitt

lavalamp · 2018-11-06T22:25:34Z

staging/src/k8s.io/apiserver/pkg/storage/value/metrics.go

@@ -41,6 +41,23 @@ var (
 		},
 		[]string{"transformation_type"},
 	)
+	transformerSuccessTotal = prometheus.NewCounterVec(


Three metrics:

total transformation counter

a counter of either successes or failures

some measure of latency, not sure the best way to format it off the top of my head.

I may be missing something in your comment.
Including the changes in this PR, we will have the following metrics:

transformation_success_total (this PR)
transformation_failures_total
transformation_latencies_microseconds
transformation_last_status (this PR)

Sorry I failed to expand the github diff. success_total + failures_total + latencies are sufficient. I'd slightly prefer if you added a total usage counter rather than a success counter.

Yes, in retrospect, I would have define a single counter - operations_total with two labels (success and failure).

Yes, in retrospect, I would have define a single counter - operations_total with two labels (success and failure).

It's not too late. If we're adding a metric here, let's add the one we want.

The current metric doesn't provide info about which resource type is being transformed (the feature is not limited just to Secret resources). Is that important to include?

Added transformation_operations_total with two labels: transformation_type and status.
I left transformation_failures_total, despite its being redundant - it may be already used.

With respect to including resource as label; how would we do it? Today, transformers, are not aware of what resource is being transformed. We could:

Infer the resource type from value.Context - it contains etcd key, which in turn contains the resource type (ex. /registry/secrets/default/dev-db-secret-0166 - second part is our resource type).

Pass resource type into TransformTo/FromStorage from the store.go.

2 would be my preference, but I'm ok with deferring that until there is demand for it

staging/src/k8s.io/apiserver/pkg/storage/value/metrics.go

immutableT · 2018-11-07T17:56:20Z

/ok-to-test

k8s-ci-robot · 2018-11-07T17:56:33Z

@immutableT: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/ok-to-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

immutableT · 2018-11-07T18:50:15Z

/ok-to-test

immutableT · 2018-11-07T23:15:30Z

@lavalamp PTAL.
I removed the transformer_last_status metric.

liggitt · 2019-01-02T19:47:55Z

staging/src/k8s.io/apiserver/pkg/storage/value/metrics.go

+		},
+		[]string{"transformation_type", "status"},
+	)
+	// Deprecated, use transformerOperationsTotal instead.


note the deprecation in the help text so callers are aware?

I followed the pattern we used in k8s.io/client-go/util/workqueue/metrics.go, and added deprecated prefix to the variable.
Also added deprecation notice to the help text.

Shouldn't the deprecation notice be added to line 59 then?

I can go either way.
Tools like Intellij/Goland provide visual indication of the deprecation when the notice is placed at the variable declaration level.

Yes, but if it is added to the help it will be output in '/metrics' endpoint which is kinda too nice, imo.

Makes sense.
Done.

liggitt · 2019-01-02T19:48:06Z

add deprecation to release note

immutableT · 2019-01-02T21:48:10Z

I assume that deprecation notice is added here as a separate PR:
https://github.com/kubernetes/website/blob/master/content/en/docs/setup/release/notes.md
after this PR is merged, correct?

immutableT · 2019-01-03T20:00:40Z

/retest

liggitt · 2019-01-16T21:33:53Z

I assume that deprecation notice is added here as a separate PR:
https://github.com/kubernetes/website/blob/master/content/en/docs/setup/release/notes.md
after this PR is merged, correct?

it goes in the PR description inside the ```release-note block

squash, then lgtm

danielqsj · 2019-03-08T01:12:35Z

staging/src/k8s.io/apiserver/pkg/storage/value/metrics.go

 		prometheus.CounterOpts{
 			Namespace: namespace,
 			Subsystem: subsystem,
 			Name:      "transformation_failures_total",
-			Help:      "Total number of failed transformation operations.",
+			Help:      "Deprecated, use transformerOperationsTotal instead. Total number of failed transformation operations.",


The deprecated description could follow this KEP , for example:

kubernetes/pkg/kubelet/dockershim/metrics/metrics.go

Line 124 in e8daa90

Help: "(Deprecated) Cumulative number of Docker operation timeout by operation type.",

What @danielqsj commented is the correct deprecation procedure. We should do it consistently in this PR.

brancz · 2019-03-11T12:57:01Z

besides some consistency on the deprecation process, this looks good from instrumentation side

immutableT · 2019-05-14T23:02:27Z

@awly PTAL

brancz · 2019-05-15T07:33:35Z

/lgtm

awly · 2019-05-15T16:27:37Z

staging/src/k8s.io/apiserver/pkg/storage/value/metrics.go

@@ -20,6 +20,10 @@ import (
 	"sync"
 	"time"

+	"google.golang.org/grpc/status"
+


remove empty line

awly · 2019-05-15T16:28:32Z

staging/src/k8s.io/apiserver/pkg/storage/value/metrics.go

@@ -106,23 +121,37 @@ func RegisterMetrics() {
 	registerMetrics.Do(func() {
 		prometheus.MustRegister(transformerLatencies)
 		prometheus.MustRegister(deprecatedTransformerLatencies)
-		prometheus.MustRegister(transformerFailuresTotal)
+


remove all the newlines here too

awly · 2019-05-15T16:31:42Z

staging/src/k8s.io/apiserver/pkg/storage/value/metrics.go

+		deprecatedTransformerLatencies.WithLabelValues(transformationType).Observe(sinceInMicroseconds(start))
+	default:
+		deprecatedTransformerFailuresTotal.WithLabelValues(transformationType).Inc()
+		st, ok := status.FromError(err)


You could just do

transformerOperationsTotal.WithLabelValues(transformationType, status.Code(err)).Inc()

outside the switch. It'll handle nil errors too.

awly · 2019-05-15T16:32:09Z

staging/src/k8s.io/apiserver/pkg/storage/value/metrics_test.go

@@ -0,0 +1,97 @@
+/*
+Copyright 2017 The Kubernetes Authors.


Still see 2017 here

awly · 2019-05-15T16:34:45Z

staging/src/k8s.io/apiserver/pkg/storage/value/metrics_test.go

+				"apiserver_storage_transformation_operations_total",
+				"apiserver_storage_transformation_failures_total",
+			},
+			want: `


Should there be latencies? Or will testutil.GatherAndCompare do a substring match?

Yes, it testuilt does will only compare the metrics explicitly requested.
Not sure if I can reliably test latencies using method, so leaving them out.

Makes sense, thanks for clarifying

immutableT · 2019-05-15T16:35:16Z

/test pull-kubernetes-e2e-gce-100-performance

immutableT · 2019-05-15T22:29:17Z

staging/src/k8s.io/apiserver/pkg/storage/value/metrics.go

@@ -20,6 +20,8 @@ import (
 	"sync"
 	"time"

+	"google.golang.org/grpc/status"


immutableT · 2019-05-15T22:29:25Z

staging/src/k8s.io/apiserver/pkg/storage/value/metrics.go

@@ -20,6 +20,10 @@ import (
 	"sync"
 	"time"

+	"google.golang.org/grpc/status"
+


immutableT · 2019-05-15T22:29:36Z

staging/src/k8s.io/apiserver/pkg/storage/value/metrics_test.go

@@ -0,0 +1,97 @@
+/*
+Copyright 2017 The Kubernetes Authors.


immutableT · 2019-05-15T22:32:13Z

staging/src/k8s.io/apiserver/pkg/storage/value/metrics_test.go

+				"apiserver_storage_transformation_operations_total",
+				"apiserver_storage_transformation_failures_total",
+			},
+			want: `


Yes, it testuilt does will only compare the metrics explicitly requested.
Not sure if I can reliably test latencies using method, so leaving them out.

immutableT · 2019-05-15T23:08:42Z

/test pull-kubernetes-node-e2e

zouyee · 2019-05-16T07:58:57Z

/lgtm

liggitt · 2019-05-17T14:51:10Z

staging/src/k8s.io/apiserver/pkg/storage/value/metrics.go

-	if err != nil {
-		transformerFailuresTotal.WithLabelValues(transformationType).Inc()
-		return
+	transformerOperationsTotal.WithLabelValues(transformationType, status.Code(err).String()).Inc()


what's the cost of status.Code(err).String(), and are the possible values bounded?

The possible values are bound to these.
I would classify the cost as negligible, see here

@liggitt PTAL

@liggitt
I ran benchmarks against the RecordTransformation function and it completes within approximately 420 ns for a generic error and in 540 ns for a status error.
With respect to the possible values, they are bound to 17 possible values, but in the context of KMS Plugin we would expect OK, Cancelled, Unknown, DeadlineExceeded, NotFound, PermissionsDenied, ResourceExhausted and FailedPrecondition.

liggitt · 2019-05-28T15:35:56Z

/approve

k8s-ci-robot · 2019-05-28T15:36:45Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: immutableT, liggitt

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~staging/src/k8s.io/apiserver/pkg/storage/OWNERS~~ [liggitt]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

tpepper · 2019-05-30T18:05:25Z

@kubernetes/sig-api-machinery-pr-reviews @kubernetes/sig-instrumentation-pr-reviews can somebody comment on the validity of the cherry pick to 1.14 here? Ie: we don't cherry pick features, so is this mismarked, or the CP invalid?

k8s-ci-robot requested review from timothysc and wojtek-t November 6, 2018 21:56

k8s-ci-robot added area/apiserver sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Nov 6, 2018

k8s-ci-robot assigned lavalamp Nov 6, 2018

k8s-ci-robot assigned liggitt Nov 6, 2018

lavalamp reviewed Nov 6, 2018

View reviewed changes

staging/src/k8s.io/apiserver/pkg/storage/value/metrics.go Outdated Show resolved Hide resolved

timothysc removed their request for review November 7, 2018 16:18

k8s-ci-robot removed the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Nov 7, 2018

liggitt reviewed Jan 2, 2019

View reviewed changes

k8s-ci-robot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Jan 16, 2019

danielqsj reviewed Mar 8, 2019

View reviewed changes

immutableT force-pushed the kube-apiserver-metrics branch from 190c9e3 to 1067d76 Compare May 6, 2019 20:36

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 6, 2019

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels May 14, 2019

immutableT force-pushed the kube-apiserver-metrics branch from 9fc8216 to 6411222 Compare May 15, 2019 00:26

k8s-ci-robot assigned brancz May 15, 2019

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 15, 2019

immutableT force-pushed the kube-apiserver-metrics branch from 6411222 to 98b1d43 Compare May 15, 2019 15:30

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 15, 2019

awly reviewed May 15, 2019

View reviewed changes

immutableT force-pushed the kube-apiserver-metrics branch from 28076e7 to 7074a4f Compare May 15, 2019 18:06

Add transformation_operations_total to transformer metrics.

90c9421

immutableT force-pushed the kube-apiserver-metrics branch from 7074a4f to 90c9421 Compare May 15, 2019 21:51

immutableT commented May 15, 2019

View reviewed changes

k8s-ci-robot assigned zouyee May 16, 2019

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 16, 2019

liggitt reviewed May 17, 2019

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 28, 2019

k8s-ci-robot merged commit 3c3c1b1 into kubernetes:master May 28, 2019

zshihang mentioned this pull request May 30, 2019

Automated cherry pick of #70715: Add transformation_operations_total to transformer metrics. #78538

Closed

Add transformation_success_total and transformation_last_status metrics. #70715

Add transformation_success_total and transformation_last_status metrics. #70715

Conversation

immutableT commented Nov 6, 2018 • edited

immutableT commented Nov 6, 2018

immutableT commented Nov 6, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

immutableT commented Nov 7, 2018

k8s-ci-robot commented Nov 7, 2018

immutableT commented Nov 7, 2018

immutableT commented Nov 7, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liggitt commented Jan 2, 2019

immutableT commented Jan 2, 2019

immutableT commented Jan 3, 2019

liggitt commented Jan 16, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brancz commented Mar 11, 2019

immutableT commented May 14, 2019

brancz commented May 15, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

immutableT commented May 15, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

immutableT commented May 15, 2019

zouyee commented May 16, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liggitt commented May 28, 2019

k8s-ci-robot commented May 28, 2019

tpepper commented May 30, 2019

immutableT commented Nov 6, 2018 •

edited