Kubelet with anonymous-auth documentation is laking

[josselin-c]

1. What kops version are you running?
   Version 1.7.1

2. What Kubernetes version are you running?
   kubectl version
   Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.4", GitCommit:"793658f2d7ca7f064d2bdf606519f9fe1229c381", GitTreeState:"clean", BuildDate:"2017-08-17T08:48:23Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"darwin/amd64"}

3. What cloud provider are you using?
   AWS

4. What commands did you run? What is the simplest way to reproduce this issue?

kops create cluster --dns-zone=lab-aws.jossctz-test.com --zones=us-east-1a,us-east-1b --name=k8s-kops.lab-aws.jossctz-test.com
kops edit cluster k8s-kops.lab-aws.jossctz-test.com
# Then I add to the spec section:
kubelet:
  anonymousAuth: false
# And finally
kops update cluster k8s-kops.lab-aws.jossctz-test.com --yes

5. What happened after the commands executed?
   The cluster isn't working. Kubectl fails connecting to the API Server: kubectl get pods returns an IO Timeout.

6. What did you expect to happen?
   My cluster is running and safe from https://github.com/kayrus/kubelet-exploit kind of attacks.

7. Please provide your cluster manifest.

apiVersion: kops/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: 2017-11-19T15:15:36Z
  name: k8s-kops.lab-aws.jossctz-test.com
spec:
  api:
    dns: {}
  authorization:
    alwaysAllow: {}
  channel: stable
  cloudProvider: aws
  configBase: s3://jossctz/k8s-kops.lab-aws.jossctz-test.com
  dnsZone: lab-aws.jossctz-test.com
  etcdClusters:
  - etcdMembers:
    - instanceGroup: master-us-east-1a
      name: a
    name: main
  - etcdMembers:
    - instanceGroup: master-us-east-1a
      name: a
    name: events
  kubelet:
    anonymousAuth: false
  kubernetesApiAccess:
  - 0.0.0.0/0
  kubernetesVersion: 1.7.10
  masterInternalName: api.internal.k8s-kops.lab-aws.jossctz-test.com
  masterPublicName: api.k8s-kops.lab-aws.jossctz-test.com
  networkCIDR: 172.20.0.0/16
  networking:
    kubenet: {}
  nonMasqueradeCIDR: 100.64.0.0/10
  sshAccess:
  - 0.0.0.0/0
  subnets:
  - cidr: 172.20.32.0/19
    name: us-east-1a
    type: Public
    zone: us-east-1a
  - cidr: 172.20.64.0/19
    name: us-east-1b
    type: Public
    zone: us-east-1b
  topology:
    dns:
      type: Public
    masters: public
    nodes: public

---

apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: 2017-11-19T15:15:36Z
  labels:
    kops.k8s.io/cluster: k8s-kops.lab-aws.jossctz-test.com
  name: master-us-east-1a
spec:
  image: kope.io/k8s-1.7-debian-jessie-amd64-hvm-ebs-2017-07-28
  machineType: m3.medium
  maxSize: 1
  minSize: 1
  role: Master
  subnets:
  - us-east-1a

---

apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: 2017-11-19T15:15:36Z
  labels:
    kops.k8s.io/cluster: k8s-kops.lab-aws.jossctz-test.com
  name: nodes
spec:
  image: kope.io/k8s-1.7-debian-jessie-amd64-hvm-ebs-2017-07-28
  machineType: t2.medium
  maxSize: 2
  minSize: 2
  role: Node
  subnets:
  - us-east-1a
  - us-east-1b

---

I want to setup a cluster with kops where pods can't talk directly to the kubelet. I think I have to set anonymousAuth: false and it should work but it doesn't.
I looked in other issues and the documentation, tried a few things but nothing worked.
Maybe the procedure should be easier to find/more explicit.

[josselin-c]

ping @gambol99 @justinsb

[chrislovecnm]

/cc ‪@bradgeesaman ‬

What is the best practice security setup?

[bgeesaman]

@josselin-c What are the full command line options that your kubelet is running with? --anonymous-auth=false is only part of the solution. You also need to ensure --client-ca-file=/path/ca.pem and --authorization-mode=Webhook. Are you missing other TLS related items?

Here https://github.com/bgeesaman/kubernetes-the-hard-way/blob/0aaf79ec93356f3afee534d67e17acca273c5d25/docs/09-bootstrapping-kubernetes-workers.md is a working kubelet command set for a prior release of Kubernetes the Hard Way as a reference.

[bgeesaman]

--anonymous-auth=false only tells kubelet to not accept anonymous API calls. So, basically, no calls will work in your cluster ATM.

Kubelet needs to use the --client-ca-file to obtain the Subject from the client token/cert to send via Webhook call against the API Server's SubjectAccessReview API to get an allow/block answer.

[gambol99]

What is the best practice security setup?

So you should definitely have anonymousAuth=false; you can get up to a lot of mischief otherwise, assuming you are not blocking the local kubelet api port from your containers

In kops, if you set --anonymous-auth=false it automatically add the --client-ca-file here and the options switched on at the kube-apiserver. Adding the authorization-mode=Webhook is a good shout though .. I don't think the componentconfig.go currently exposes this ...

[josselin-c]

Thanks for the pointers, I better understand what options I have to set now:

Kubelet: 
  anonymous-auth=false
  authorization-mode=Webhook
  client-ca-file=/var/lib/kubernetes/ca.pem
APIServer: 
  nothing?

It doesn't seem possible to set authorization-mode=Webhook on the kubelet configuration though. I don't see a matching attribute in the KubeletConfigSpec object (in componentconfig.go)

[gambol99]

correct ... authorization-mode=Webhook isn't available for configuration at the moment. Adding that one requires a little more thought as it would changes to RBAC ... @justinsb @chrislovecnm ??

[bgeesaman]

@josselin-c Can you manually edit that setting for your kubelet on one worker node (SSH in, edit, restart kubelet) and see if that node still functions correctly (pods schedule, you can kubectl exec and kubectl logs that pod, etc)? If not, can you post the RBAC logs that show any deny entries related to kubelet on that node?

Over the next few weeks, I'll be looking into these specifics myself, but this was the process I took when submitting the PR I linked above to Kubernetes the Hard Way to validate the configuration.

[josselin-c]

Okay, here is what I did:
Edit /etc/sysconfig/kubelet so it looks like that (added --anonymous-auth=false --authorization-mode=Webhook --client-ca-file=/var/lib/kubernetes/ca.pem):

root@ip-172-20-36-15:/home/admin# cat /etc/sysconfig/kubelet
DAEMON_ARGS="--allow-privileged=true --cgroup-root=/ --cloud-provider=aws --cluster-dns=100.64.0.10 --cluster-domain=cluster.local --enable-debugging-handlers=true --eviction-hard=memory.available<100Mi,nodefs.available<10%,nodefs.inodesFree<5%,imagefs.available<10%,imagefs.inodesFree<5% --hostname-override=ip-172-20-36-15.ec2.internal --kubeconfig=/var/lib/kubelet/kubeconfig --network-plugin-mtu=9001 --network-plugin=kubenet --node-labels=kubernetes.io/role=node,node-role.kubernetes.io/node= --non-masquerade-cidr=100.64.0.0/10 --pod-infra-container-image=gcr.io/google_containers/pause-amd64:3.0 --pod-manifest-path=/etc/kubernetes/manifests --register-schedulable=true --require-kubeconfig=true --v=2 --cni-bin-dir=/opt/cni/bin/ --cni-conf-dir=/etc/cni/net.d/ --network-plugin-dir=/opt/cni/bin/ --anonymous-auth=false --authorization-mode=Webhook --client-ca-file=/var/lib/kubernetes/ca.pem"

Create the /var/lib/kubernetes/ca.pem file with the certificate-authority-data field of the /var/lib/kubelet/kubeconfig file:

Made the /etc/sysconfig/kubelet imutable so it isn't replaced next boot: chattr +i /etc/sysconfig/kubelet

Then: reboot

After that, the node is marked as Running, I can schedule pods on it but I cannot exec into it:

$ kubectl exec -ti debian-2251103498-p5sqp bash
error: unable to upgrade connection: Unauthorized

Maybe I set the wrong CA? /var/lib/kubernetes/ca.pem didn't exists before I created it.

[bgeesaman]

Ok, I spun up a 1.7.1 cluster with everything as defaults and hand-edited things until they worked. This is not a final solution per se, but rather a map on how to get to a potentially working destination.

NOTE: Do not deploy these changes to a cluster you care about. I enable RBAC here, and I guarantee that I'm breaking other services via missed RBAC policies.

On the workers, go into /var/lib/kubelet.

1. Extract out of kubeconfig the ca.pem, kubelet.cert, and kubelet.key files into /var/lib/kubelet (copy each section, base64 decode to file)
2. chmod 640 kubelet.*
3. Add --authorization-mode=Webhook --anonymous-auth=false --client-ca-file=/var/lib/kubelet/ca.pem --tls-cert-file=/var/lib/kubelet/kubelet.cert --tls-private-key-file=/var/lib/kubelet/kubelet.key to /etc/sysconfig/kubelet
4. Restart the kubelet: systemctl daemon-reload && systemctl restart kubelet

On the master, edit /etc/kubernetes/manifests/kube-apiserver.manifest:

1. Change authorization-mode from AlwaysAllow to RBAC
--authorization-mode=RBAC
2. Add the following options
   --kubelet-client-certificate=/srv/kubernetes/kubelet.cert --kubelet-client-key=/srv/kubernetes/kubelet.key --audit-log-path=- (The audit-log-path option is optional, but very useful for debugging RBAC. They will show up in /var/log/kube-apiserver.log. NOT good for disk space in production, FYI.)
3. Copy kubelet.key and kubelet.cert from a worker into /var/lib/kubernetes on this master and chmod 640 kubelet.*
4. Restart the kubelet: systemctl daemon-reload && systemctl restart kubelet

Run $ kubectl edit clusterrolebinding system:node
and edit it to look like this:

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:node
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:node
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:nodes

Run $ kubectl edit clusterroles system:node
and add:

- apiGroups:
  - ""
  resources:
  - nodes/proxy
  verbs:
  - create
  - get

You should now be able to perform exec and log actions again.

So, I know that isn't a "add this option to kops in a yaml and go" solution, but it does outline some of the work needed to make this function as intended.

[bgeesaman]

Issue #1231 is what is causing the additional RBAC items to be necessary (instead of following the naming convention that gets the permissions automatically applied)

[chrislovecnm]

/area security

[chrislovecnm]

@josselin-c this issue should be coverred by getting #1231 working in kops. Agreed? If so can we close this as a duplicate?

[josselin-c]

Probably.
For info, here is what I'm using on my cluster with flannel (no network policies) to "fix" the issue:
EDIT: This solution doesn't work, see bgeesaman comments bellow

kind: DaemonSet
apiVersion: extensions/v1beta1
metadata:
  namespace: kube-system
  name: blackhole-kubelet
  labels:
    app: blackhole-kubelet
spec:
  template:
    metadata:
      labels:
        app: blackhole-kubelet
    spec:
      hostPID: true
      containers:
        - name: blackhole-kubelet
          image: gcr.io/google-containers/startup-script:v1
          securityContext:
            privileged: true
          env:
          - name: STARTUP_SCRIPT
            value: |
              #! /bin/bash
              while true; do
                iptables-save | grep INPUT | grep -q "KUBELET-BLACKHOLE"
                if [ $? -ne 0 ]; then
                   echo "Missing kubelet Blackhole rule, adding it"
                   iptables -I INPUT -s 100.64.0.0/10 -m comment --comment "KUBELET-BLACKHOLE: block kubelet access from pods" -m tcp  -p tcp --dport 10250 -j REJECT --reject-with icmp-port-unreachable
                fi
                sleep 60
              done

[bgeesaman]

@josselin-c Clever! Are you using this successfully in a cluster running calico/weave/other?

[josselin-c]

On clusters with calico/weave I'd look into NetworkPolicies, I had to use a DaemonSet because flannel doesn't support them.

[bgeesaman]

Ah, that makes sense. IME, Calico monitors “foreign” iptables rules and removes them automatically. In cases where networkpolicy doesn’t yet support egress filtering, rules like these can be a useful stopgap if they can be made to “stick”.

[bgeesaman]

I take that back. Kops 1.7.1 (k8s 1.7.10) with calico does not modify the INPUT chain in this case, so this policy stays in place and works to block pods hitting the local node's kubelet port. However, it does NOT prevent pods on one worker node from crossing over and hitting other worker nodes and more importantly, the master node's kubelet port. This is because by that time, it's NAT-ed out via the eth0 IP.

This is the shortened output of a simple shell script that I run inside a pod to see what it can see/do:

# ./audit.sh 
...snip...
 7 - Access Kubelet on local host (https://172.20.50.151:10250/runningpods/): False
 8 - Access Kubelet on another worker host (https://172.20.53.168:10250/runningpods/): True
 9 - Access Kubelet on master host (https://172.20.57.132:10250/runningpods/): True
...snip...

Notice how # 7 is blocked but # 8 and # 9 still succeed.

This is what the traffic from the audit pod (100.97.190.131/32) on worker (172.20.50.151) going to the master (172.20.57.132) looks like on its way out:

# ssh 172.20.50.151
# tcpdump -ni eth0 port 10250
16:40:38.679005 IP 172.20.50.151.40304 > 172.20.57.132.10250: Flags [S], seq 2002535152, win 29200, options [mss 1460,sackOK,TS val 2247126 ecr 0,nop,wscale 9], length 0
16:40:38.679326 IP 172.20.57.132.10250 > 172.20.50.151.40304: Flags [S.], seq 3126813096, ack 2002535153, win 26847, options [mss 8961,sackOK,TS val 2247315 ecr 2247126,nop,wscale 9], length 0
16:40:38.679365 IP 172.20.50.151.40304 > 172.20.57.132.10250: Flags [.], ack 1, win 58, options [nop,nop,TS val 2247126 ecr 2247315], length 0

A crude, but certainly workable stop-gap is to edit the outbound rules on the worker security-group rule from:

| All traffic | All | All | 0.0.0.0/0 |

to be something like:

| Custom TCP Rule | TCP | 10251 - 65535 | 0.0.0.0/0 |  
| Custom TCP Rule | TCP | 0 - 10249     | 0.0.0.0/0 |  
| All UDP         | UDP | 0 - 65535     | 0.0.0.0/0 |  
| All ICMP - IPv4 | All | N/A           | 0.0.0.0/0 |

Now, the run looks like:

...snip...
 7 - Access Kubelet on local host (https://172.20.50.151:10250/runningpods/): False
 8 - Access Kubelet on another worker host (https://172.20.53.168:10250/runningpods/): False
 9 - Access Kubelet on master host (https://172.20.57.132:10250/runningpods/): False
...snip...

Of course, these workarounds aren't needed with 1.8.x+ and CNI plugins (like calico) that support egress networkpolicy and egress policies on namespaces that block this pod-to-node mgmt traffic. Even once the kubelet is configured to perform authn/authz via web hook, you still don't want those ports exposed. Defense in depth, etc.

[josselin-c]

Thanks for the review, indeed it wasn't enough to block traffic from pods CIDR.
Here is another try at fixing the issue while staying with flannel-only networking:
https://gist.github.com/josselin-c/3002e9bac8be27305b579ba6650ad8da

[bgeesaman]

Again, very clever!

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close