Auth{n,z} delegation for Webhooks / endpoints (metrics, healthz, etc...) #1385
Description
In newer Kubernetes versions its possible configure API servers to authenticate when making calls to Mutating / Validating Webhooks. For this to work, the webhook has to have several filters added to it. When a request is received, it would do TokenReview / SubjectAccessReview to authenticate and authorize the request.
Ideally you should be able to secure:
- the
/metricsendpoint - the
/healtzendpoint - the webhooks endpoint
by providing authentication and authorization configuration for each of those endpoints.
Today for some of those endpoints this is possible, but for others not possible at all. The k8s.io/apiserver package has lots of the security features in there - for example to be able to provide my own server that would have authentication configuration that be added to all http handlers.
For my webhook, I had to do lots of hacking:
package options
import (
"github.com/spf13/pflag"
"k8s.io/apimachinery/pkg/runtime"
apiserver "k8s.io/apiserver/pkg/server"
genericoptions "k8s.io/apiserver/pkg/server/options"
)
type Options struct {
SecureServing *genericoptions.SecureServingOptionsWithLoopback
Authentication *genericoptions.DelegatingAuthenticationOptions
Authorization *genericoptions.DelegatingAuthorizationOptions
}
func NewOptions() *Options {
// reuse the K8S defaults
recommended := genericoptions.NewRecommendedOptions("", runtime.NoopDecoder{})
opts := &Options{
SecureServing: recommended.SecureServing,
Authentication: recommended.Authentication,
Authorization: recommended.Authorization,
}
opts.Authentication.RemoteKubeConfigFileOptional = true
opts.Authorization.RemoteKubeConfigFileOptional = true
return opts
}
func (o *Options) AddFlags(fs *pflag.FlagSet) {
o.SecureServing.AddFlags(fs)
o.Authentication.AddFlags(fs)
o.Authorization.AddFlags(fs)
}
// ApplyTo adds Options to the server configuration.
func (o *Options) ApplyTo(server *Config) error {
if err := o.SecureServing.ApplyTo(&server.SecureServing, nil); err != nil {
return err
}
if err := o.Authentication.ApplyTo(&server.Authentication, server.SecureServing, nil); err != nil {
return err
}
if err := o.Authorization.ApplyTo(&server.Authorization); err != nil {
return err
}
return nil
}
func (o *Options) Validate() []error {
errors := []error{}
errors = append(errors, o.SecureServing.Validate()...)
errors = append(errors, o.Authentication.Validate()...)
errors = append(errors, o.Authorization.Validate()...)
return errors
}
// Config has all the context to run an webhook
type Config struct {
Authentication apiserver.AuthenticationInfo
Authorization apiserver.AuthorizationInfo
SecureServing *apiserver.SecureServingInfo
}Then in my main I had to manually register the webhook with more boiler plate
conf := &options.Config{}
options.NewOptions().ApplyTo(conf)
...
// MyWebhook.Build creates the chain from https://github.com/kubernetes/kubernetes/blob/cd21a1240a5f07e9b6c5f48e72cfd354b5183d34/cmd/kube-scheduler/app/server.go#L224-L236
mgr.GetWebhookServer().Register("/my-webhook",
(&custom.MyWebhook{}).Build(conf.Authentication.Authenticator, conf.Authorization.Authorizer, conf.Authentication.APIAudiences))I was not able to directly change the behavior and add auth{n,z} filter for /healthz and /metrics endpoints. One thing that could be done to mitigate the situation is to change the GetWebhookServer() to an interface so I can create a manager wrapper that would return my webhook server implementation that would add filters out of the box.
Another alternative is to use kube-rbac-proxy to solve those problems, but it might also be helpful to have this functionality build-in.