Authenticated/Authorized access to kubelet API

[liggitt]

Description

The kubelet API gives access to a wide variety of resources.

Allow authenticating requests to the kubelet API using any of:

• x509 client certs
• bearer token (validated using the TokenReview API)
• anonymous authentication (default in 1.0-1.4)

Allow authorizing requests to the kubelet API using one of:

• SubjectAccessReview API
• AlwaysAllow (default in 1.0-1.4)

Progress Tracker

• Before Alpha
  • Write and maintain draft quality doc
    • During development keep a doc up-to-date about the desired experience of the feature and how someone can try the feature in its current state. Think of it as the README of your new feature and a skeleton for the docs to be written before the Kubernetes release. https://github.com/liggitt/kubernetes.github.io/blob/kubelet-authnz/docs/admin/kubelet-authentication-authorization.md
  • Design Approval
    • Design Proposal: Proposal: kubelet authentication/authorization kubernetes#32518
    • My Shepherd is: @liggitt
    • My Secondary Contact Point is: @deads2k @erictune
  • Write (code + tests + docs) then get them merged.
    • Revert "Revert "Allow webhook authenticator to use TokenReviewsInterface"" kubernetes#32597 - refactor authentication webhook
    • Allow webhook authorizer to use SubjectAccessReviewInterface kubernetes#34071 - refactor authorization webhook
    • kubelet authn/authz kubernetes#34381 - kubelet authn/authz
    • Code needs to be disabled by default.
    • Minimal testing
    • Minimal docs
      • Update kubelet authentication/authorization docs website#1684
      • cc @kubernetes/docs on docs PR
      • cc @kubernetes/feature-reviewers on this issue to get approval before checking this off
    • Update release notes (release notes in kubelet authn/authz kubernetes#34381)
• Before Beta
  • Testing is sufficient for beta
  • User docs with tutorials
    • Update kubelet authentication/authorization docs website#1684
    • Updated walkthrough / tutorial in the docs repo: kubernetes/kubernetes.github.io
    • cc @kubernetes/docs on docs PR
    • cc @kubernetes/feature-reviewers on this issue to get approval before checking this off
• Before Stable
  • docs/proposals/foo.md moved to docs/design/foo.md
    • cc @kubernetes/feature-reviewers on this issue to get approval before checking this off
  • Soak, load testing
  • detailed user docs and examples
    • cc @kubernetes/docs
    • cc @kubernetes/feature-reviewers on this issue to get approval before checking this off

FEATURE_STATUS: IN_DEVELOPMENT

[liggitt]

cc @kubernetes/sig-auth

[deads2k]

cc @kubernetes/sig-auth

Just noting approval for the feature.

[philips]

@liggitt @deads2k are y'all planning on doing this work? I am cross-tracking this on the SIG node task list.

[liggitt]

@philips yes, I'll be doing the work

[jessfs]

Just wondering, will this be in 1.5? Trying to determine if our team needs to invest the time into securing these communications via SSH and firewall rules to prevent https://github.com/kayrus/kubelet-exploit, or if we can just hold off for a bit and utilize our existing TLS client certs once this lands.

[deads2k]

Just wondering, will this be in 1.5?

Yes, we are targeting 1.5.

[jessfs]

@deads2k awesome, thanks!!

[kevin-wangzefeng]

/cc @kubernetes/huawei

[idvoretskyi]

Does this feature target alpha for 1.5?

[deads2k]

Does this feature target alpha for 1.5?

@idvoretskyi at least alpha

@erictune @liggitt This mirrors the mechanism we've used in OpenShift since 1.0. Want to call it beta?

[liggitt]

I'd be comfortable with beta. It is built on top of two beta APIs, and has been tested. The remaining work is automated load testing and default enablement in the various install/setup methods

[erictune]

No objection to beta. Ask node team too?

On Thu, Nov 17, 2016 at 5:29 AM, Jordan Liggitt notifications@github.com
wrote:

I'd be comfortable with beta. It is built on top of two beta APIs, and has
been tested. The remaining work is automated load testing and default
enablement in the various install/setup methods

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#89 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHuudnmBmvnWB6MaZFzXHqy96__HdUdGks5q_FbRgaJpZM4J65Mz
.

[derekwaynecarr]

I am a lead on @kubernetes/sig-node and agree this should be beta.

[spiffxp]

I'm removing the team/SIG-Node label given that SIG-Auth is listed as the owner in the 1.5 feature spreadsheet; if we're using labels to describe areas of overlap then we need something else in these issues to identify owner

[idvoretskyi]

@liggitt can you confirm that this item targets stable in 1.6?

[liggitt]

@liggitt can you confirm that this item targets stable in 1.6?

Yes

[liggitt]

keeping in beta status until the TokenReview and SubjectAccessReview APIs move to stable (now targeting 1.7)

[grodrigues3]

@liggitt @idvoretskyi moving to next milestone then

[joantune]

Hi guys!

So, what's the status on this? I saw some PR for docs which seems to have changed.

Having a secure kubernetes cluster right out of kubeadm 'box' would be great. On a sidenote can one use calico or any of the Networking and Network Policy addons for this? Clearly AFAIK VXLAN based ones are insufficient as VXLAN doesn't provide security (please correct me if i'm wrong) but I'm thinking something like Calico could help? would a properly configured Calico be enough to make the These connections are not currently safe to run over untrusted and/or public networks warning from the apiserver -> nodes, pods, and services section that lives here obsolete?

[luxas]

Having a secure kubernetes cluster right out of kubeadm 'box' would be great

That is the case. kubeadm uses the latest stable security features for pretty much every release.

Regarding the other questions you have, I don't think they are very relevant to this feature. This feature is about locking down the kubelet API endpoint, not network (Pod2Pod, Node2Node) communication.

@liggitt I guess we could close this. Stable in v1.6 which was released some time ago. Most providers (like kubeadm) enable this by default.

[joantune]

Regarding the other questions you have, I don't think they are very relevant to this feature. This feature is about locking down the kubelet API endpoint, not network (Pod2Pod, Node2Node) communication.

Locking it down as in an access control applied to the level of the kubelet API endpoint itself, right? not preventing access from it, because I can still pretty much do a wget --no-check-certificate https://<ip>:10250 on the nodes etc.
It would be great to have a way to only access this through ssh tunnels or perhaps HTTP client based certificates, no? although ssh tunnels sound more secure (as the port wouldn't be available from the outside)

In no way I have considered the implications/work of such a feature given Kubernetes architecture. Simply by my book no port to connect to available is better than having it available

[luxas]

Please read https://github.com/kubernetes/kubernetes.github.io/blob/master/docs/admin/kubelet-authentication-authorization.md

The kubelet port at https://<ip>:10250 can with this feature (in the kubeadm case) only be accessed by client certificates signed by the cluster CA with O=system:masters, which basically means API Servers are the only ones that can access the kubelet ports by default.

wget --no-check-certificate https://:1025

That will yield an Unauthorized response. If you try to use a client cert that is not in the system:masters organization you will get Forbidden as the return.

[joantune]

That will yield an Unauthorized response. If you try to use a client cert that is not in the system:masters organization you will get Forbidden as the return.

🤔 I was pretty sure I got a 404 instead of a 401 from running that wget with an out of the box kubeadm 1.7.1 setup cluster. I guess I can always change the config, but it would be nice if kubeadm did it for us from the start perhaps (?)

[ericchiang]

@joantune @luxas can you open issues against kubeadm for this? This issue is only to track the feature implemented in the kubelet.

[joantune]

By the way, the documentation is ok detailing what are the concepts of Nodes, Pods and Clusters, and how they interact with each other, however, I found no reference on how these interact with the host's world, i.e. it would be nice to see some diagrams or similar on how the several services and concepts (pods, nodes, services, processes [i.e. kubectl, kubeproxy kubeadm]) interact with the host and its native network.

I have been looking quite thoroughly at the docs yet I have no clear picture of how that looks like in my head. Making it clearer would lower the barrier to learn it and use it.

Can anyone point me to a diagram/doc? I'm asking because I want to know exactly what should I expose or shouldn't network wise.

For instance, imagine that I have an equivalent to the Amazon's VPCs, what should be exposed to the outside what doesn't? how does a service get exposed? (i.e. is there any kubernetes process for which all the traffic flows through [e.g. kubeproxy] or does it expose the pod's port directly) would it be ok to have kubectl only accessible internally (if i'm willing to do an SSH tunnel to one of the machines). Would that still allow Kubernetes to work well?

These things would help sysadmins to have a clear picture if their network and cluster configuration are secure or not and confident that they are exposing the least they should

[davidopp]

@joantune The best reference I've found for some of the questions you asked is this video:
https://www.youtube.com/watch?v=y2bhV81MfKQ

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[luxas]

keeping in beta status until the TokenReview and SubjectAccessReview APIs move to stable (now targeting 1.7)

@liggitt I'm pretty sure we can close this now as many deployments secure the kubelet API OOTB, and the APIs are v1/stable already.

[liggitt]

@liggitt I'm pretty sure we can close this now as many deployments secure the kubelet API OOTB, and the APIs are v1/stable already.

agree, closing