Allow Kubelet API bearer token authentication

[dghubble]

Feature Request

Feature

Today, Kubelet bearer token authentication is not permitted. Allow Kubelet bearer token authentication and enable Webhook authorization (required, as the default would be AlwaysAllow) to enable giving out more fine grained RBAC permissions related to nodes. One example use-case is reducing the Prometheus addon's RBAC privileges from node/proxy to node/metrics.

This impacts several areas where we need to be careful to avoid regressions:

• kubectl commands
• Prometheus addon
• Potentially metric-server addon

More context: #191

[dghubble]

Evaluating PR #191 showed simply adding --authentication-token-webhook and --authorization-mode=Webhook breaks kubectl logs, exec, port-forward, and other essentials (unusual). Some digging shows this seems to be a symptom of not using a special TLS organization in apiserver certificate generation.

Here's how it works.

Today

Today, Typhoon (along with bootkube and likely Tectonic)) do not enable Kubelet bearer token authentication. kubectl commands like exec, logs, port-forward authenticate with the apiserver and then apiserver uses its X509 --kubelet-client-certificate and --kubelet-client-key key to authenticate to a particular Kubelet. Requests succeed because the apiserver's certificate authenticates and the Kubelet defaults to AlwaysAllow authorization.

Verify the apiserver can present its TLS cert and do whatever it likes:

kubectl exec -it kube-apiserver-sr8z2 -n kube-system /bin/bash
curl --cert /etc/kubernetes/secrets/apiserver.crt --key /etc/kubernetes/secrets/apiserver.key  http://NODE-INTERNAL-IP:10250/metrics -k

This design is reasonable and has served well for a while. No reason to worry. The apiserver X509 certificate is generated upfront and kept safe, since it allows all-access to kubelets.

Kubelet Bearer Token Auth

Experiment with Kubelet bearer token authentication. Launch a cluster and enable Kubelet bearer token auth on one of the worker nodes by editing the unit file and restarting the kubelet.

--authentication-token-webhook
--authorization-mode=Webhook

This opens up the Kubelet to accepting bearer tokens (e.g. from a pod's service account), so we switch the default authorization from AlwaysAllow to Webhook. As a side-effect, that means the X509 client certificate requests must also be authorized now.

Try to get the logs of a pod on that node with kubectl or exec into the apiserver to repeat the verification above.

curl --cert /etc/kubernetes/secrets/apiserver.crt --key /etc/kubernetes/secrets/apiserver.key  http://NODE-INTERNAL-IP:10250/metrics -k
Forbidden (user=kube-apiserver, verb=get, resource=nodes, subresource=proxy)

While the apiserver.crt and apiserver.key are valid and authenticate just fine, they are not authorized. This is surprising. Enabling Kubelet authorization shows the apiserver is not actually authorized to perform log, exec, etc.

Tracking this back the apiserver TLS generation, I notice Typhoon, bootkube, and Tectonic all generate apiserver certificates with an ORG: kube-master:

• https://github.com/poseidon/terraform-render-bootkube/blob/master/tls-k8s.tf#L65
• https://github.com/kubernetes-incubator/bootkube/blob/master/pkg/asset/tls.go#L87
• https://github.com/coreos/tectonic-installer/blob/master/modules/tls/kube/api.tf#L13

I believe the special org is supposed to be system:masters. I launched a Typhoon cluster with modifications to the apiserver's ORG to system:masters. Now when I enable kublet bearer token auth, it does not break kubectl log, exec, port-forward, etc.

I'm not sure where ORG kube-master came from, but I suspect its wrong.

Note: In #191, someone claimed Tectonic was using kubelet token auth. I'm not 100% (since I can't see the proprietary bits and its a bit different), but given Tectonic is also using kube-master, I think its probably also not using token auth (or else they'd have kubectl issues).

[ericchiang]

When you enable:

--authorization-mode=Webhook

that causes the kubelet to make an authorization query against the API server to determine if the subject can perform actions against the kubelet API. The following ClusterRole should satisfy all kubelet authorization queries:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: kubelet-api
rules:
- apiGroups: [""]
  resources: ["nodes/stats", "nodes/metrics", "nodes/log", "nodes/spec", "nodes/proxy"]
  verbs: ["get", "list", "update", "patch", "delete"]

Then bind it to whatever the API server's client certs authenticate as. X509 certs authenticate by CN -> username, and O -> group:

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: kubelet-api
subjects:
- kind: User
  name: kube-apiserver
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: kubelet-api
  apiGroup: rbac.authorization.k8s.io

[ericchiang]

The reason system:masters works is because it maps to an admin RBAC role. That works too, but wanted to explain why it works a little.

[dghubble]

Awesome! Thanks for adding that.

I'm thinking to go with the kube-master -> system:masters switch. Since its the X509 certificate for the apiserver itself, it seems like it has a pretty legitimate claim to it.

[dghubble]

Added in #216

[aknuds1]

Does this mean that Prometheus Operator/kube-prometheus now works with Typhoon as I set out to solve with #191?

[dghubble]

Perhaps. You'd need to check their docs or try it to be sure, Prometheus Operator isn't something I'm going to be using or validating.

The motive was further improving the security of the Prometheus addon's manifests that are designed for usage with Typhoon. Great that it helps Prometheus Operator too. We recommend the Prometheus addon however. Compare the Prometheus addon's ClusterRole with the Prometheus Operator's ClusterRole and compare the flexibility and features of each. Choose what's best for you - these are addons atop a cluster after all and its your choice.