SELinux vs Kubernetes volumes

[kadel]

SELinux inside ADB is blocking using emptyDir and hostPath volumes and passing secrets to containers as volumes

emptyDir and hostPath volumes

If you create pod with volume, where volume is either emptyDir or hostPath,
SELinux is blocking access to that directory inside container.

example

busybox.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  containers:
  - image: busybox
    name: busybox
    command:
      - sleep
      - "36000"
    volumeMounts:
      - name: storage
        mountPath: /storage
  volumes:
    - name: storage
      hostPath:
        path: /tmp/storage 

kubectl create -f busybox.yaml

kubectl exec -it busybox /bin/sh
/ # touch /storage/asdf
touch: /storage/asdf: Permission denied

I have to manually change context of /tmp/storage on host to get that working

chcon -Rt svirt_sandbox_file_t /tmp/storage

Same issue is with emptyDir volumes. This situation is a little bit complicated because emptyDir volumes are created in path containing pod uid /var/lib/kubelet/pods/4dd7b77b-8228-11e5-b8db-525400e09276/volumes/kubernetes.io~empty-dir/

Kubernetes secrets

SELinux is also blocking access to secret volumes so you can not pass secrets from kubernetes to containers.

example:

secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: mysecret
type: Opaque
data:
  password: dmFsdWUtMg0K
  username: dmFsdWUtMQ0K

busybox.yaml

apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  containers:
  - image: busybox
    name: busybox
    command:
      - sleep
      - "36000"
    volumeMounts:
      - name: storage
        mountPath: /storage
      - name: secret
        mountPath: /secret

  volumes:
    - name: storage
      hostPath:
        path: /tmp/storage 
    - name: secret
      secret:
        secretName: mysecret

kubectl  create  -f secret.yaml
kubectl  create  -f busybox.yaml

now you should be able to access /secret/username and /secret/password inside container

# kubectl exec -it busybox /bin/sh
# ls /secret/
ls: /secret/username: Permission denied
ls: /secret/password: Permission denied

to get this manually working is a little bit pain:

# first you need to get pod id
PODID=$(kubectl  get pod busybox -o template --template="{{ .metadata.uid }}")
# thank you can change security context of volume on host
chcon -Rt svirt_sandbox_file_t /var/lib/kubelet/pods/$PODID/volumes/kubernetes.io~secret

# kubectl exec -it busybox /bin/sh
# cat /secret/*
value-2
value-1

It is quite pain to work with emptyDir and secrets as volumes inside ADB, you have to always think of chcon. When you stop pod it is recreated with new uid, so you have to chcon secrets and emptyDir volumes again :-(

I don't know if this is Kubernetes problem or something with SELinux setup inside ADB.

I think that in hostDir case it is ok to require manual chcon, but with emptyDir and with secrets, that should be automatic.

My idea is to automatically label everything in /var/lib/kubelet/pods/*/volumes/ svirt_sandbox_file_t.

[praveenkumar]

My idea is to automatically label everything in /var/lib/kubelet/pods/*/volumes/ svirt_sandbox_file_t.

I think this can be handled by k8s SELinux rules (may be have those entry to spec file) but it's should be discussed with @rhatdan to understand if there any drawback.

[rhatdan]

If you label everything in this directory with this label then (::svirt_sandbox_file_t:s0) then all containers will be able to read write all content there. If you use the "Z" when volume mounting into a container then docker will relabel the content to something that is private to the container.

[ediskandarov]

Are there any progress on this issue.
This issue blocks using kubernetes with Atomic Host in production.

[praveenkumar]

Proposal is already available to kubernetes GitHub to make sure SELinux context (https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/selinux.md ) should be correct and it is used in OpenShift downstream work (kubernetes/kubernetes#9844). PR say it's already merged. I will test latest branch and check if that something we want for ADB.

[praveenkumar]

I tried latest kubernetes (update-testing repo) in fedora-23 and still empty directory have permission issue.

# cat busybox.yaml
apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  containers:
  - image: busybox
    name: busybox
    command:
      - sleep
      - "36000"
    volumeMounts:
      - name: storage
        mountPath: /storage
  volumes:
    - name: storage
      emptyDir: {}

# kubectl get pods
NAME      READY     STATUS    RESTARTS   AGE
busybox   1/1       Running   0          40s

# kubectl exec -it busybox /bin/sh
/ # cd /storage/
/storage # touch hello
touch: hello: Permission denied

But as per kubernetes/kubernetes#15883 it should be resolved for fedora so I will discuss it with @pmorie

[praveenkumar]

I had discussion with @pmorie and looks like labeling /var/lib/kubectl/pods with ::svirt_sandbox_file_t:s0 will be good enough for this issue because latest version of k8s (kubernetes/kubernetes@1d352a1) have Z added for emptydir which relabel it to different context as @rhatdan said.

Now we have to make sure if k8s shipped by ADB have this fix or not.

[LalatenduMohanty]

@praveenkumar ADB/CDK is based on CentOS/RHEL. So it needs to be fixed in RHEL and ADB/CDK rebuild will get it.

We should file a bug against RHEL and then can put a workaround in the kickstart file.

[praveenkumar]

@LalatenduMohanty yes I filled issue against RHEL https://bugzilla.redhat.com/show_bug.cgi?id=1298568

[praveenkumar]

Some more observation after checking if stock k8s has auto labeling of shared volumes but looks like it still lack of it.

Plain docker container

 # docker run -d -v /root/hello/:/strorage:rw,Z --name test docker.io/busybox /bin/sleep 36000
980411a0c1ee9c4689b4ecde78810c0bc712e14d88e04f205aa57a2fdb77e830
# ls -Zl .
total 8
-rw-------. 1 system_u:object_r:admin_home_t:s0 root root 4283 Apr  5 02:13 anaconda-ks.cfg
drwxr-xr-x. 2 system_u:object_r:svirt_sandbox_file_t:s0:c499,c672 root root    6 Apr  6 02:30 hello
# docker inspect test | grep -i mount
    "MountLabel": "system_u:object_r:svirt_sandbox_file_t:s0:c499,c672", 
   "Mounts": [
        {
            "Source": "/root/hello",
            "Destination": "/strorage",
            "Mode": "rw,Z",
            "RW": true
        }

k8s container

Using same template as I pointed above (used by @kadel also)

# kubectl create -f busybox.yaml
# kubectl get pods
NAME      READY     STATUS    RESTARTS   AGE
busybox   1/1       Running   1          18h
# docker ps
7ccc35980a91        busybox                                                      "sleep 36000"        8 hours ago         Up 8 hours                              k8s_busybox.ed662816_busybox_default_76979d50-fb2b-11e5-8b91-52540099d993_a9d80ec0
d716252fea3e        registry.access.redhat.com/rhel7/pod-infrastructure:latest   "/pod"               18 hours ago        Up 18 hours                             k8s_POD.ae8ee9ac_busybox_default_76979d50-fb2b-11e5-8b91-52540099d993_c2e36ba7
# docker inspect test | grep -i mount
   "MountLabel": "system_u:object_r:svirt_sandbox_file_t:s0:c499,c672",
   "Mounts": [
        {
            "Source": "/var/lib/kubelet/pods/76979d50-fb2b-11e5-8b91-52540099d993/volumes/kubernetes.io~empty-dir/storage",
            "Destination": "/storage",
            "Mode": "",
            "RW": true
        },

# ls -Zl /var/lib/kubelet/pods/76979d50-fb2b-11e5-8b91-52540099d993/volumes/kubernetes.io~empty-dir
total 0
drwxrwxrwx. 2 system_u:object_r:svirt_sandbox_file_t:s0 root root 27 Apr  5 12:11 storage

Conclusion

So k8s does relabel it but only issue I am worried about Mount mode.