Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

keep the status error from envelope service #82543

Merged
merged 1 commit into from Sep 12, 2019

Conversation

@shihan9
Copy link
Contributor

commented Sep 10, 2019

Change-Id: I8263c4673d5f57617acf315c7af6ebe5aacd9c7c

this PR will propagate the status error from envelope provider to metrics so that can record the useful error code for better monitoring.

i would like to treat this as a bug, because it blinds us from differentiating errors we got from envelope provider. if envelope provider returns FailedPrecondition, right now will classify it as Unknown.

What type of PR is this?
/kind bug

none
@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

commented Sep 10, 2019

Hi @shihan9. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@shihan9

This comment has been minimized.

Copy link
Contributor Author

commented Sep 10, 2019

/assign @liggitt
/cc @immutableT

@shihan9

This comment has been minimized.

Copy link
Contributor Author

commented Sep 10, 2019

/ok-to-test

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

commented Sep 10, 2019

@shihan9: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/ok-to-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@immutableT

This comment has been minimized.

Copy link
Contributor

commented Sep 10, 2019

/ok-to-test

return b.Bytes()
out, err := b.Bytes()
if err != nil {
return nil, status.Error("failed to get encrypted data")

This comment has been minimized.

Copy link
@liggitt

liggitt Sep 10, 2019

Member

why are we masking the returned error here?

This comment has been minimized.

Copy link
@shihan9

shihan9 Sep 10, 2019

Author Contributor

should be status.Error(codes.Internal, "failed to get encrypted data")

# HELP apiserver_storage_transformation_operations_total [ALPHA] Total number of transformations.
# TYPE apiserver_storage_transformation_operations_total counter
apiserver_storage_transformation_operations_total{status="OK",transformation_type="from_storage",transformer_prefix="k8s:enc:kms:v1:"} 1
apiserver_storage_transformation_operations_total{status="OK",transformation_type="to_storage",transformer_prefix="k8s:enc:kms:v1:"} 1

This comment has been minimized.

Copy link
@liggitt

liggitt Sep 10, 2019

Member

cc @logicalhan on metrics changes

This comment has been minimized.

Copy link
@logicalhan

logicalhan Sep 10, 2019

Contributor

There are no actual metrics changes.. this is just tests. Was this test not testing the thing it was supposed to, previously?

@@ -83,7 +84,7 @@ func (t *envelopeTransformer) TransformFromStorage(data []byte, context value.Co
var encKey cryptobyte.String
s := cryptobyte.String(data)
if ok := s.ReadUint16LengthPrefixed(&encKey); !ok {
return nil, false, fmt.Errorf("invalid data encountered by envelope transformer: failed to read uint16 length prefixed data")
return nil, false, status.Error(codes.Internal, "failed to read prefixed data")

This comment has been minimized.

Copy link
@liggitt

liggitt Sep 10, 2019

Member

this is a bit misleading... it wasn't actually a grpc error, right?

This comment has been minimized.

Copy link
@immutableT

immutableT Sep 10, 2019

Contributor

That is correct, this is not a grpc error.
However, we label transformers' metrics based on status.Code.
If we don't assign a code here, this error will be classified as Unknown, which is probably not ideal.

@shihan9 shihan9 force-pushed the shihan9:master branch from 80a69e5 to 425f112 Sep 10, 2019

@@ -83,7 +84,7 @@ func (t *envelopeTransformer) TransformFromStorage(data []byte, context value.Co
var encKey cryptobyte.String
s := cryptobyte.String(data)
if ok := s.ReadUint16LengthPrefixed(&encKey); !ok {
return nil, false, fmt.Errorf("invalid data encountered by envelope transformer: failed to read uint16 length prefixed data")
return nil, false, status.Error(codes.Internal, "failed to read prefixed data")

This comment has been minimized.

Copy link
@immutableT

immutableT Sep 10, 2019

Contributor

Why change the existing error message?

}
transformer, err = t.addTransformer(encKey, key)
if err != nil {
return nil, false, err
return nil, false, status.Error(codes.Internal, "failed to add transformer")

This comment has been minimized.

Copy link
@immutableT

immutableT Sep 10, 2019

Contributor

Same as above, keep the existing message.

This comment has been minimized.

Copy link
@shihan9

shihan9 Sep 10, 2019

Author Contributor

PTAL

return transformer.TransformFromStorage(encData, context)

out, stale, err := transformer.TransformFromStorage(encData, context)
if err != nil {

This comment has been minimized.

Copy link
@immutableT

immutableT Sep 10, 2019

Contributor

What if an error came from a kms-plugin with some specific status?
Would not this make every error from a plugin to have a status of Internal?
I think we need to use status.FromError here.

This comment has been minimized.

Copy link
@shihan9

shihan9 Sep 10, 2019

Author Contributor

this transformer is not envelope transformer. what we get from kms-plugin is from envelopeService.

i'll remove these as suggested by liggit@

return b.Bytes()
out, err := b.Bytes()
if err != nil {
return nil, status.Error("failed to get encrypted data")

This comment has been minimized.

Copy link
@immutableT

immutableT Sep 10, 2019

Contributor

What should be the status here?

This comment has been minimized.

Copy link
@shihan9

shihan9 Sep 10, 2019

Author Contributor

done

@liggitt

This comment has been minimized.

Copy link
Member

commented Sep 10, 2019

in several places, this change masks the returned error. can we avoid that, while still providing an appropriate error code to the metrics?

@shihan9 shihan9 force-pushed the shihan9:master branch from 425f112 to 19660f3 Sep 10, 2019

@k8s-ci-robot k8s-ci-robot added size/M and removed size/L labels Sep 10, 2019

@shihan9 shihan9 changed the title change envelope transformer to return status error for better monitoring keep the status error from envelope service Sep 10, 2019

@shihan9

This comment has been minimized.

Copy link
Contributor Author

commented Sep 10, 2019

you are right. i should not misuse grpc/status. PTAL @liggitt

@logicalhan
Copy link
Contributor

left a comment

/lgtm

There are no real metrics metrics changes here. The metrics changes are all test-output related.

@shihan9

This comment has been minimized.

Copy link
Contributor Author

commented Sep 10, 2019

/retest

@shihan9 shihan9 requested a review from liggitt Sep 10, 2019

@@ -94,13 +94,14 @@ func (t *envelopeTransformer) TransformFromStorage(data []byte, context value.Co
value.RecordCacheMiss()
key, err := t.envelopeService.Decrypt(encKey)
if err != nil {
return nil, false, fmt.Errorf("error while decrypting key: %q", err)
return nil, false, err

This comment has been minimized.

Copy link
@immutableT

immutableT Sep 10, 2019

Contributor

We should add a comment that explains why we don't wrap (add more information) to the error message.
It may be better to be explicit and use status.FromError - this way we make it explicit that our contract is to produce an error with Status.

This comment has been minimized.

Copy link
@shihan9

shihan9 Sep 10, 2019

Author Contributor

status.FromError is producing status.Status, not an error.
status.FromError(err).Err() == err

This comment has been minimized.

Copy link
@mikedanese

mikedanese Sep 11, 2019

Member

The %q should definitely be a %v. As of go1.13 which we are upgrading to imminently, you can call https://godoc.org/errors#As. Are we using status.FromErr yet?

This comment has been minimized.

Copy link
@shihan9

shihan9 Sep 11, 2019

Author Contributor

yep, https://godoc.org/errors#As can be used to unwrap the error before we record the metric.

no, we are not using status.FromError.

@shihan9 shihan9 force-pushed the shihan9:master branch from 19660f3 to ab4044a Sep 10, 2019

@k8s-ci-robot k8s-ci-robot removed the lgtm label Sep 10, 2019

change envelope transformer to return status error for better monitoring
Change-Id: I8263c4673d5f57617acf315c7af6ebe5aacd9c7c

@shihan9 shihan9 force-pushed the shihan9:master branch from ab4044a to cba4353 Sep 10, 2019

@liggitt

This comment has been minimized.

Copy link
Member

commented Sep 12, 2019

/priority important-longterm
/lgtm
/approve

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

commented Sep 12, 2019

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: liggitt, shihan9

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot merged commit 41b0ae6 into kubernetes:master Sep 12, 2019

25 checks passed

cla/linuxfoundation shihan9 authorized
Details
pull-kubernetes-bazel-build Job succeeded.
Details
pull-kubernetes-bazel-test Job succeeded.
Details
pull-kubernetes-conformance-image-test Skipped.
pull-kubernetes-conformance-kind-ipv6 Skipped.
pull-kubernetes-cross Skipped.
pull-kubernetes-dependencies Job succeeded.
Details
pull-kubernetes-e2e-gce Job succeeded.
Details
pull-kubernetes-e2e-gce-100-performance Job succeeded.
Details
pull-kubernetes-e2e-gce-alpha-features Skipped.
pull-kubernetes-e2e-gce-csi-serial Skipped.
pull-kubernetes-e2e-gce-device-plugin-gpu Job succeeded.
Details
pull-kubernetes-e2e-gce-iscsi Skipped.
pull-kubernetes-e2e-gce-iscsi-serial Skipped.
pull-kubernetes-e2e-gce-storage-slow Skipped.
pull-kubernetes-godeps Skipped.
pull-kubernetes-integration Job succeeded.
Details
pull-kubernetes-kubemark-e2e-gce-big Job succeeded.
Details
pull-kubernetes-local-e2e Skipped.
pull-kubernetes-node-e2e Job succeeded.
Details
pull-kubernetes-node-e2e-containerd Job succeeded.
Details
pull-kubernetes-typecheck Job succeeded.
Details
pull-kubernetes-verify Job succeeded.
Details
pull-publishing-bot-validate Skipped.
tide In merge pool.
Details

@k8s-ci-robot k8s-ci-robot added this to the v1.17 milestone Sep 12, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.