Ensure webhook service routing resolves kubernetes.default.svc correctly #62649

liggitt · 2018-04-16T13:29:36Z

Going through the normal endpoint resolve path isn't correct in multi-master scenarios

The auth wrapper is pulling from LoopbackClientConfig, the service resolver should do the same

Fixes the kubernetes.default.svc loopback service resolution to use a loopback configuration.

liggitt · 2018-04-16T13:29:53Z

/assign @deads2k

deads2k · 2018-04-16T13:32:02Z

/lgtm

lavalamp · 2018-04-16T17:00:21Z

Please test these things if you want them to keep working:

/lgtm cancel

I think this is an awful hack to work around not having an identity for apiserver. e.g. clearly this can't work for an aggregated apiserver. Or am I misunderstanding something?

liggitt · 2018-04-16T17:14:36Z

I think this is an awful hack to work around not having an identity for apiserver

the only service the apiserver can make authentication assumptions about is the kubernetes/default service, and only because it is serving it and controls authentication for it.

e.g. clearly this can't work for an aggregated apiserver

that's why this is only used in kube-apiserver. aggregated api servers running on top of the kube control plane use default dns resolution and in-cluster auth without incident.

lavalamp · 2018-04-16T17:39:14Z

the only service the apiserver can make authentication assumptions about is the kubernetes/default service, and only because it is serving it and controls authentication for it.

A concrete example request that exercises the path would help a lot.

liggitt · 2018-04-17T05:09:40Z

test added (verified it failed on master with an unresolveable address timeout), three additional bugs fixed. most integration tests do not actually use the real apiserver construction methods, so wouldn't have exercised the code that was fixed.

The closest integration test to reality I found was this one:

kubernetes/test/integration/examples/apiserver_test.go

Lines 99 to 140 in bf3cda6

    
           kubeAPIServerOptions := options.NewServerRunOptions() 
        
           kubeAPIServerOptions.SecureServing.Listener = listener 
        
           kubeAPIServerOptions.SecureServing.BindAddress = net.ParseIP("127.0.0.1") 
        
           kubeAPIServerOptions.SecureServing.ServerCert.CertDirectory = certDir 
        
           kubeAPIServerOptions.InsecureServing.BindPort = 0 
        
           kubeAPIServerOptions.Etcd.StorageConfig.ServerList = []string{framework.GetEtcdURL()} 
        
           kubeAPIServerOptions.ServiceClusterIPRange = *defaultServiceClusterIPRange 
        
           kubeAPIServerOptions.Authentication.RequestHeader.UsernameHeaders = []string{"X-Remote-User"} 
        
           kubeAPIServerOptions.Authentication.RequestHeader.GroupHeaders = []string{"X-Remote-Group"} 
        
           kubeAPIServerOptions.Authentication.RequestHeader.ExtraHeaderPrefixes = []string{"X-Remote-Extra-"} 
        
           kubeAPIServerOptions.Authentication.RequestHeader.AllowedNames = []string{"kube-aggregator"} 
        
           kubeAPIServerOptions.Authentication.RequestHeader.ClientCAFile = proxyCACertFile.Name() 
        
           kubeAPIServerOptions.Authentication.ClientCert.ClientCA = clientCACertFile.Name() 
        
           kubeAPIServerOptions.Authorization.Modes = []string{"RBAC"} 
        
           completedOptions, err := app.Complete(kubeAPIServerOptions) 
        
           if err != nil { 
        
           	t.Fatal(err) 
        
           } 
        
           tunneler, proxyTransport, err := app.CreateNodeDialer(completedOptions) 
        
           if err != nil { 
        
           	t.Fatal(err) 
        
           } 
        
           kubeAPIServerConfig, sharedInformers, versionedInformers, _, _, _, err := app.CreateKubeAPIServerConfig(completedOptions, tunneler, proxyTransport) 
        
           if err != nil { 
        
           	t.Fatal(err) 
        
           } 
        
           // Adjust the loopback config for external use (external server name and CA) 
        
           kubeAPIServerClientConfig := rest.CopyConfig(kubeAPIServerConfig.GenericConfig.LoopbackClientConfig) 
        
           kubeAPIServerClientConfig.CAFile = path.Join(certDir, "apiserver.crt") 
        
           kubeAPIServerClientConfig.CAData = nil 
        
           kubeAPIServerClientConfig.ServerName = "" 
        
           kubeClientConfigValue.Store(kubeAPIServerClientConfig) 
        
           kubeAPIServer, err := app.CreateKubeAPIServer(kubeAPIServerConfig, genericapiserver.NewEmptyDelegate(), sharedInformers, versionedInformers) 
        
           if err != nil { 
        
           	t.Fatal(err) 
        
           } 
        
           if err := kubeAPIServer.GenericAPIServer.PrepareRun().Run(wait.NeverStop); err != nil { 
        
           	t.Fatal(err) 
        
           }

In the test commit in this PR, I started to extract the test setup to something that:

gives you a one-call invocation to get a running apiserver
lets you tweak run options or config prior to starting
makes use of the actual config and server-building functions that kube-apiserver uses

https://github.com/kubernetes/kubernetes/blob/596e4c6c48316a808ff3c16afdf4c756d572e204/test/integration/examples/webhook_test.go#L43-L62

lavalamp · 2018-04-17T05:19:11Z

Thank you!

Will look tomorrow if no one beats me to it.

liggitt · 2018-04-17T12:00:08Z

/retest

deads2k · 2018-04-17T12:41:22Z

staging/src/k8s.io/client-go/transport/transport.go

@@ -52,7 +52,7 @@ func New(config *Config) (http.RoundTripper, error) {
 // TLSConfigFor returns a tls.Config that will provide the transport level security defined
 // by the provided Config. Will return nil if no transport level security is requested.
 func TLSConfigFor(c *Config) (*tls.Config, error) {
-	if !(c.HasCA() || c.HasCertAuth() || c.TLS.Insecure) {
+	if !(c.HasCA() || c.HasCertAuth() || c.TLS.Insecure || len(c.TLS.ServerName) > 0) {


if tls.Insecure is true, why don't we just return nil, nil?

anything that deviates from the defaults must return a custom tls.Config

setting Insecure=true or a custom ServerName requires a non-default tls.Config

deads2k · 2018-04-17T12:45:28Z

Question on the TLS bit. Integration test looks like it is testing the wiring more than the code since kube integration doesn't run real servers, but sure.

liggitt · 2018-04-20T15:43:13Z

any other comments?

deads2k · 2018-04-20T15:49:17Z

/lgtm

k8s-ci-robot · 2018-04-20T15:49:25Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: deads2k, liggitt

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~cmd/kube-apiserver/OWNERS~~ [deads2k,liggitt]
~~staging/src/k8s.io/apiserver/OWNERS~~ [deads2k,liggitt]
~~staging/src/k8s.io/client-go/OWNERS~~ [deads2k,liggitt]
~~staging/src/k8s.io/kube-aggregator/OWNERS~~ [deads2k]
~~test/OWNERS~~ [deads2k,liggitt]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2018-04-20T16:31:06Z

New changes are detected. LGTM label has been removed.

liggitt · 2018-04-20T16:31:11Z

rebased

fejta-bot · 2018-04-20T19:31:52Z

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel comment for consistent failures.

k8s-github-robot · 2018-04-20T22:05:57Z

/test all [submit-queue is verifying that this PR is safe to merge]

k8s-github-robot · 2018-04-20T22:34:12Z

Automatic merge from submit-queue (batch tested with PRs 50899, 62649). If you want to cherry-pick this change to another branch, please follow the instructions here.

k8s-ci-robot · 2018-04-20T22:36:51Z

@liggitt: The following tests failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
pull-kubernetes-e2e-gce	`d421aff`	link	`/test pull-kubernetes-e2e-gce`
pull-kubernetes-integration	`d421aff`	link	`/test pull-kubernetes-integration`
pull-kubernetes-e2e-kops-aws	`d421aff`	link	`/test pull-kubernetes-e2e-kops-aws`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

…9-upstream-release-1.10 Automatic merge from submit-queue. Automated cherry pick of #62649: Ensure service routing resolves kubernetes.default.svc Cherry pick of #62649 on release-1.10. #62649: Ensure service routing resolves kubernetes.default.svc

munnerz · 2018-04-25T11:52:04Z

Is this likely to be also accepted into release-1.9 if cherry picked? Not sure what the usual convention is for which releases something should be backported to.

liggitt · 2018-04-27T00:29:56Z

Is this likely to be also accepted into release-1.9 if cherry picked? Not sure what the usual convention is for which releases something should be backported to.

picked to 1.9 in #63219

k8s-ci-robot requested review from caesarxuchao and mikedanese April 16, 2018 13:29

k8s-ci-robot assigned deads2k Apr 16, 2018

liggitt mentioned this pull request Apr 16, 2018

Automated cherry pick of #62649: Ensure service routing resolves kubernetes.default.svc #62654

Merged

k8s-ci-robot assigned lavalamp Apr 16, 2018

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 16, 2018

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Apr 17, 2018

liggitt force-pushed the loopback-routing branch from 596e4c6 to ffe61f1 Compare April 17, 2018 06:03

deads2k reviewed Apr 17, 2018

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 20, 2018

liggitt added 5 commits April 20, 2018 12:26

Ensure service routing resolves kubernetes.default.svc correctly

fe23fa3

distinguish custom dialers in transport cache

d45fbce

ensure tls server name is used in transport

6f65742

Honor existing CA bundle and TLS server name in webhook client

54c883f

loopback webhook integration test

d421aff

liggitt force-pushed the loopback-routing branch from ffe61f1 to d421aff Compare April 20, 2018 16:31

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 20, 2018

liggitt added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 20, 2018

k8s-github-robot merged commit 9c25da6 into kubernetes:master Apr 20, 2018

munnerz mentioned this pull request Apr 25, 2018

Add validating webhook and webhook tls autoconfiguration cert-manager/cert-manager#478

Merged

liggitt deleted the loopback-routing branch April 25, 2018 20:02

tamalsaha mentioned this pull request Aug 20, 2018

Cannot create deployment. Stash admission policy is preventing stashed/stash#557

Closed

munnerz mentioned this pull request Feb 28, 2019

cainjector: support injecting apiserver ca cert-manager/cert-manager#1420

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ensure webhook service routing resolves kubernetes.default.svc correctly #62649

Ensure webhook service routing resolves kubernetes.default.svc correctly #62649

liggitt commented Apr 16, 2018 •

edited by deads2k

Loading

liggitt commented Apr 16, 2018

deads2k commented Apr 16, 2018

lavalamp commented Apr 16, 2018

liggitt commented Apr 16, 2018

lavalamp commented Apr 16, 2018

liggitt commented Apr 17, 2018 •

edited

Loading

lavalamp commented Apr 17, 2018

liggitt commented Apr 17, 2018

deads2k Apr 17, 2018

liggitt Apr 17, 2018

deads2k commented Apr 17, 2018

liggitt commented Apr 20, 2018

deads2k commented Apr 20, 2018

k8s-ci-robot commented Apr 20, 2018

k8s-ci-robot commented Apr 20, 2018

liggitt commented Apr 20, 2018

fejta-bot commented Apr 20, 2018

k8s-github-robot commented Apr 20, 2018

k8s-github-robot commented Apr 20, 2018

k8s-ci-robot commented Apr 20, 2018 •

edited

Loading

munnerz commented Apr 25, 2018

liggitt commented Apr 27, 2018

Ensure webhook service routing resolves kubernetes.default.svc correctly #62649

Ensure webhook service routing resolves kubernetes.default.svc correctly #62649

Conversation

liggitt commented Apr 16, 2018 • edited by deads2k Loading

liggitt commented Apr 16, 2018

deads2k commented Apr 16, 2018

lavalamp commented Apr 16, 2018

liggitt commented Apr 16, 2018

lavalamp commented Apr 16, 2018

liggitt commented Apr 17, 2018 • edited Loading

lavalamp commented Apr 17, 2018

liggitt commented Apr 17, 2018

deads2k Apr 17, 2018

Choose a reason for hiding this comment

liggitt Apr 17, 2018

Choose a reason for hiding this comment

deads2k commented Apr 17, 2018

liggitt commented Apr 20, 2018

deads2k commented Apr 20, 2018

k8s-ci-robot commented Apr 20, 2018

k8s-ci-robot commented Apr 20, 2018

liggitt commented Apr 20, 2018

fejta-bot commented Apr 20, 2018

k8s-github-robot commented Apr 20, 2018

k8s-github-robot commented Apr 20, 2018

k8s-ci-robot commented Apr 20, 2018 • edited Loading

munnerz commented Apr 25, 2018

liggitt commented Apr 27, 2018

liggitt commented Apr 16, 2018 •

edited by deads2k

Loading

liggitt commented Apr 17, 2018 •

edited

Loading

k8s-ci-robot commented Apr 20, 2018 •

edited

Loading