⚠️ webhook redesign for generic case #323

DirectXMan12 · 2019-02-11T23:35:28Z

While implementing a webhook, I found some tech debt and rough corners around the webhook code. This is an attempt to smooth some of that over.

k8s-ci-robot · 2019-02-11T23:35:35Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: DirectXMan12

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [DirectXMan12]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

DirectXMan12 · 2019-02-11T23:35:35Z

cc @mengqiy

DirectXMan12 · 2019-02-11T23:36:11Z

(built on top of #300 )

DirectXMan12 · 2019-02-11T23:36:59Z

/assign @mengqiy

droot

Love this change. I have couple of comments. I will take another look from the perspective of adding conversion-webhook-support tomorrow.

droot · 2019-02-12T21:42:06Z

example/main.go

-		os.Exit(1)
-	}
+	hookServer.Register("/mutate-pods", webhook.Admission{&podAnnotator{}})
+	hookServer.Register("/validate-pods", webhook.Admission{&podValidator{}})


Looks very neat, easier to follow. This means path is now decoupled from the webhook definition.

droot · 2019-02-12T21:43:21Z

example/mutatingwebhook.go

 // podAnnotator adds an annotation to every incoming pods.
-func (a *podAnnotator) Handle(ctx context.Context, req types.Request) types.Response {
+func (a *podAnnotator) Handle(ctx context.Context, req admission.Request) admission.Response {


+1 on readability admission.{Request, Response} as http.{Request, Response}

droot · 2019-02-12T21:53:50Z

pkg/webhook/admission/http.go

 )

-var admissionv1beta1scheme = runtime.NewScheme()
-var admissionv1beta1schemecodecs = serializer.NewCodecFactory(admissionv1beta1scheme)
+var admissionScheme = runtime.NewScheme()


Does this scheme needs user defined types ? (basically, should we be injected from the manager ?)

This decoder only need to understand AdmissionReviewRequest. So IMO no need to be injected.

droot · 2019-02-12T21:57:46Z

pkg/webhook/admission/multi.go

+func (hs multiMutating) Handle(ctx context.Context, req Request) Response {
+	patches := []jsonpatch.JsonPatchOperation{}
+	for _, handler := range hs {
+		resp := handler.Handle(ctx, req)


looks like these handlers are independent, so can probably be invoked concurrently ?

maybe, but I almost want to deprecate this.

maybe, but I almost want to deprecate this.

Reasons?

droot · 2019-02-12T22:08:49Z

pkg/webhook/server.go

+func instrumentedHook(path string, hookRaw http.Handler) http.Handler {
+	return http.HandlerFunc(func(resp http.ResponseWriter, req *http.Request) {
+		startTS := time.Now()
+		defer metrics.RequestLatency.WithLabelValues(path).Observe(time.Now().Sub(startTS).Seconds())


We need to wrap it with func{} otherwise the args time.Now().Sub(startTS).Seconds() will be evaluated immediately reporting almost no latency :)

Good catch!
The deferred call's arguments are evaluated immediately, but the function call is not executed until the surrounding function returns.
Ref: https://tour.golang.org/flowcontrol/12

mengqiy

A lot of improvement!

I have several comment inlined.

mengqiy · 2019-02-12T21:34:07Z

example/main.go

 		Port:    9876,
 		CertDir: "/tmp/cert",
 	})
 	if err != nil {
 		entryLog.Error(err, "unable to create a new webhook server")
 		os.Exit(1)
 	}
+	mgr.Add(as)


I don't think this will work, since mgr.Add() actually does the deps injection.
It should be done after registering all the handlers.
Thus, I proposed using Complete method to finish the setup.

Start does the deps injection, really. Add just records the setFields method for later.

mengqiy · 2019-02-12T22:04:39Z

pkg/webhook/server.go

-type Server struct {
-	// Name is the name of server
-	Name string
+	// TODO(directxman12): should we make the mux configurable?


Is there a use case in your head that the user may want to configure mux?

if you want to serve on an existing handler more easily. It's not clear that there's a concrete usecase, but I can imagine someone saying something at some points.

(e.g. your comment below about Handle)

(e.g. your comment below about Handle)

We probably should make webhookMux field configurable.

mengqiy · 2019-02-12T22:06:26Z

pkg/webhook/server.go

-	registry map[string]Webhook
+	// webhookMux is the multiplexer that handles different webhooks.
+	webhookMux *http.ServeMux
+	// hooks keep track of all registered webhooks for dependency injection,


nit: do we want to use hooks or webhooks here?

comment still doesn't match field name

mengqiy · 2019-02-12T22:09:08Z

pkg/webhook/server.go

 	}

 	return nil
 }

-// Handle registers a http.Handler for the given pattern.
-func (s *Server) Handle(pattern string, handler http.Handler) {


The original intend was to give the user flexibility to add additional handlers. e.g. statusz, healthz etc.

right, but they shouldn't be doing it on the webhook server. They should do it elsewhere, or we should let people pass in their own mux.

example/mutatingwebhook.go

example/validatingwebhook.go

pkg/webhook/server.go

mengqiy · 2019-02-12T22:22:56Z

pkg/webhook/admission/http.go

@@ -98,14 +89,6 @@ func (wh *Webhook) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 }

 func (wh *Webhook) writeResponse(w io.Writer, response Response) {
-	if response.Result.Code != 0 {
-		if response.Result.Code == http.StatusOK {
-			metrics.TotalRequests.WithLabelValues(wh.Path, "true").Inc()


It seems metrics TotalRequests has been dropped.
Is it not useful?

It's not clear how useful it is (accepting or rejecting is a normal occurrence -- you really probably want to see errors and error codes if you're trying to alert, etc), and it was difficult to work in the new code structure. There may be a way to implement it generically, but I figured it was fine for a follow-up PR + the scream test.

DirectXMan12 · 2019-02-20T04:10:07Z

@mengqiy this is ready for more review. I'll write a better description tomorrow. Still want to follow it up with a better story around generating patches and returning errors, perhaps.

droot · 2019-02-20T18:39:44Z

Looks good to me. Will defer it to @mengqiy for the final LGTM.

mengqiy

Ha, 20 commits now.
Overall looks good.

move webhook self installer to CT as generator

The 1st commit should be gone after the rebase. Why it is still there?

mengqiy · 2019-02-20T21:42:22Z

pkg/webhook/admission/decode.go

@@ -34,6 +36,21 @@ func NewDecoder(scheme *runtime.Scheme) (*Decoder, error) {

 // Decode decodes the inlined object in the AdmissionRequest into the passed-in runtime.Object.
 func (d *Decoder) Decode(req Request, into runtime.Object) error {
+	// NB(directxman12): there's a bug/weird interaction between decoders and


This looks like a workaround until apimachinery folks fix the bug.
We can probably link to the upstream issue.

ack, need to file the issue

mengqiy · 2019-02-20T21:45:04Z

pkg/webhook/example_test.go

+		Handler: admission.HandlerFunc(func(ctx context.Context, req AdmissionRequest) AdmissionResponse {
+			return Patched("some changes",
+				JSONPatchOp{Operation: "add", Path: "/metadata/annotations/access", Value: "granted"},
+				JSONPatchOp{Operation: "add", Path: "/metadata/annotations/access", Value: "granted"},


It is strange to have a duplicate JSONPatch in the example, isn't it?

mengqiy · 2019-02-20T21:56:36Z

example/mutatingwebhook.go

 // podAnnotator implements inject.Client.
 // A client will be automatically injected.
-var _ inject.Client = &podAnnotator{}


Why removing this?

It doesn't really add much, and necessitates an extra import. I can add it back in if we really want it, but I think the comment suffices here.

mengqiy · 2019-02-20T21:58:30Z

pkg/controller/controllerutil/controllerutil.go

@@ -22,7 +22,8 @@ import (
 	"reflect"

 	"k8s.io/apimachinery/pkg/api/errors"
-	"k8s.io/apimachinery/pkg/apis/meta/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	v1 "k8s.io/apimachinery/pkg/apis/meta/v1"


duplicate package?

mengqiy · 2019-02-20T22:00:07Z

pkg/internal/webhookgenerator/admission/webhook.go

@@ -0,0 +1,105 @@
+/*


I believe everything under pkg/internal/webhookgenerator is rebase artifacts and should not exist.

mengqiy · 2019-02-20T23:40:39Z

pkg/webhook/admission/http_test.go

+	invoked        bool
+	fn             func(context.Context, Request) Response
+	decoder        *Decoder
+	injectedString string


Is it actually tested anywhere?

yep, pkg/webhook/admission/webhook_test.go

This makes sure to run go vet on the examples and the alias file, and turns goimports back on.

We had goimports turned off. When turned back on, it complained about a bunch of stuff. This fixes that.

The example was still using the old list options arguments style (second argument instead of last argument), which made it fail to compile. This also makes sure we run go test on the whole project, instead of just pkg/...

The webhook server logic was not handling injection correctly -- it implemented injection for a couple of specific fields, rather than receiving and dealing with an injectfunc. This fixes that. As a side effect, it also removes the handle to manager and client from the webhook server, since those aren't used by the server, and should thus be injected from the manager.

This flattens down the webhook package structure, removing dedicated `types` packages in favor of having things in the relevant places. To do this, the inject interface definitions for admission were also moved to the admission controller location, but they actually make some amount of sense living there. It also renames and restructures a couple of the types for code clarity (e.g. WebhookTypeMutating to MutatingWebhook) or to follow the convention of other types in CR (e.g. Making wrapped types from core Kubernetes nested fields).

We don't have anywhere that we actually make use of decoder as an interface, so switch to just have a concrete implementation.

This extracts out the multi-handler support into separate interfaces. That simplifies the code, makes it easier to test directly, and allows us to drop the now-extraneous Type field. This simultaneously removes the builder (for now) since it doesn't actually make anything simpler. Along the way, we also refactor some common functionality out into a helper on reponse itself as opposed to being run by different bits of the webhook.

This cleans up tests to more closely follow CR style, favoring behavior-style tests as opposed to pure unit tests.

This renames a few ambigously named fields in the webhook server, removes some extraneous fields and methods, and removes the unnecessary constructor for Server.

It was only being used for registering metrics, and that can be done with the path instead.

This removes the path functionality from admission webhooks, since it's not strictly needed. Relevant metrics are moved up to the webhook server, and path is added to register. This also introduces a couple of helpers that should make code a bit clearer when writing admission webhooks (Allowed, Denied, Patched), and an alias file to avoid importing both webhook and webhook/admission.

This ensures that decoders (which are no longer constructed in the manager) are properly injected into webhook handlers. The webhook itself receives a scheme, which it uses to construct a decoder.

Due to a weird interaction between the unstructured decoder (which demands APIVersion and Kind fields) and the API server (which fails to set those fields on the embedded object in admission requests), we can't use the normal decoders, nor can we call json.Unmarshal directly on the unstructured (since it implements an unmarshaller that calls back into the decoder). This detects unstructured objects, and does the right thing for them.

This moves the examples to an actual Go example file, so that we can be sure that they actually compile.

It's not used anywhere any more.

It's the same name used in alias.go, and it produces less-verbose code without loss of readability.

The were incorrect (would not have compiled), are mostly duplicates of examples/*webhook.go, and can't be easily checked for being up-to-date.

This gives each webhook a log, so that serving errors can be tied to a particuluar webhook.

mengqiy · 2019-02-21T22:45:51Z

/lgtm

Nothing was missing from the tree (due to transitive deps), just missing from being required by Gopkg.lock.

mengqiy · 2019-02-21T23:02:39Z

/lgtm

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 11, 2019

k8s-ci-robot requested review from droot and pwittrock February 11, 2019 23:35

k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Feb 11, 2019

DirectXMan12 changed the title ~~[WIP] ⚠️ webhook refactor and redesign~~ [WIP] ⚠️ webhook refactor and redesign (for generic webhooks) Feb 11, 2019

DirectXMan12 changed the title ~~[WIP] ⚠️ webhook refactor and redesign (for generic webhooks)~~ [WIP] ⚠️ webhook redesign for generic case Feb 11, 2019

k8s-ci-robot assigned mengqiy Feb 11, 2019

DirectXMan12 mentioned this pull request Feb 12, 2019

⚠️ split webhook server and manifest generation #300

Merged

droot reviewed Feb 12, 2019

View reviewed changes

mengqiy reviewed Feb 12, 2019

View reviewed changes

mengqiy mentioned this pull request Feb 14, 2019

⚠️ Simplify webhook wiring with Defaulter and Validator interfaces #328

Merged

DirectXMan12 force-pushed the tmp/webhook-decouple branch from d4693e2 to 18b49ef Compare February 20, 2019 04:08

DirectXMan12 changed the title ~~[WIP] ⚠️ webhook redesign for generic case~~ ⚠️ webhook redesign for generic case Feb 20, 2019

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 20, 2019

DirectXMan12 force-pushed the tmp/webhook-decouple branch 2 times, most recently from a25f015 to 8623a9b Compare February 20, 2019 04:22

mengqiy reviewed Feb 20, 2019

View reviewed changes

DirectXMan12 added 5 commits February 21, 2019 13:10

Go vet entire pkg, turn goimports back on

ab62d27

This makes sure to run go vet on the examples and the alias file, and turns goimports back on.

Make everything pass goimports

f0ec881

We had goimports turned off. When turned back on, it complained about a bunch of stuff. This fixes that.

Fix example_test.go to use new list options

eee0406

The example was still using the old list options arguments style (second argument instead of last argument), which made it fail to compile. This also makes sure we run go test on the whole project, instead of just pkg/...

DirectXMan12 added 3 commits February 21, 2019 14:22

Remove extraneous decoder interface

6dbde68

We don't have anywhere that we actually make use of decoder as an interface, so switch to just have a concrete implementation.

Clean up tests to follow standards

66be336

This cleans up tests to more closely follow CR style, favoring behavior-style tests as opposed to pure unit tests.

DirectXMan12 force-pushed the tmp/webhook-decouple branch from 5978194 to be3d3ac Compare February 21, 2019 22:22

DirectXMan12 added 10 commits February 21, 2019 14:43

Clean up webhook server

9a7de90

This renames a few ambigously named fields in the webhook server, removes some extraneous fields and methods, and removes the unnecessary constructor for Server.

Remove the extraneous name field from webhooks

f46c199

It was only being used for registering metrics, and that can be done with the path instead.

Construct and inject decoder into webhook handlers

1859301

This ensures that decoders (which are no longer constructed in the manager) are properly injected into webhook handlers. The webhook itself receives a scheme, which it uses to construct a decoder.

Move webhook examples to an actual Go example file

e08eee7

This moves the examples to an actual Go example file, so that we can be sure that they actually compile.

Remove pkg/internal/admission

65a93df

It's not used anywhere any more.

Rename ErrorResponse to Errored

1a4bbe1

It's the same name used in alias.go, and it produces less-verbose code without loss of readability.

Remove inline admission examples

a866ee9

The were incorrect (would not have compiled), are mostly duplicates of examples/*webhook.go, and can't be easily checked for being up-to-date.

Add per-webhook logging

34a4153

This gives each webhook a log, so that serving errors can be tied to a particuluar webhook.

DirectXMan12 force-pushed the tmp/webhook-decouple branch from be3d3ac to 60b01e4 Compare February 21, 2019 22:44

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 21, 2019

Ensure the right deps are in Gopkg.log

286b0b4

Nothing was missing from the tree (due to transitive deps), just missing from being required by Gopkg.lock.

DirectXMan12 force-pushed the tmp/webhook-decouple branch from 60b01e4 to 286b0b4 Compare February 21, 2019 22:59

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 21, 2019

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 21, 2019

k8s-ci-robot merged commit 58a08d8 into kubernetes-sigs:master Feb 21, 2019

This was referenced Mar 6, 2019

Webhook: Need more injection Interface in webhook server #309

Closed

Decouple webhook and manager packages #190

Closed

mcristina422 mentioned this pull request Dec 5, 2019

Webhook Metric Request Total Unused #712

Closed

⚠️ webhook redesign for generic case #323

⚠️ webhook redesign for generic case #323

Conversation

DirectXMan12 commented Feb 11, 2019 • edited

k8s-ci-robot commented Feb 11, 2019

DirectXMan12 commented Feb 11, 2019

DirectXMan12 commented Feb 11, 2019

DirectXMan12 commented Feb 11, 2019

droot left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mengqiy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DirectXMan12 commented Feb 20, 2019

droot commented Feb 20, 2019

mengqiy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mengqiy commented Feb 21, 2019

mengqiy commented Feb 21, 2019

DirectXMan12 commented Feb 11, 2019 •

edited