Init container security context extended to containers

[sylr]

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

What happened:

Created a deployment with an initContainer which has a privileged security context and this privileged context extended to the container of the deployment.

What you expected to happen:

That the security context defined for initContainers does not extend to containers.

How to reproduce it (as minimally and precisely as possible):

Deployment

apiVersion: apps/v1
kind: StatefulSet
metadata:
  generation: 1
  labels:
    app: docker-elasticsearch
    release: docker-elasticsearch
    storage: docker-elasticsearch
  name: docker-elasticsearch
spec:
  podManagementPolicy: OrderedReady
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: docker-elasticsearch
      release: docker-elasticsearch
      storage: docker-elasticsearch
  serviceName: internal
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: docker-elasticsearch
        release: docker-elasticsearch
        storage: docker-elasticsearch
    spec:
      containers:
      - image: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/docker-elasticsearch:0.0.0.12
        imagePullPolicy: Always
        name: pod
        ports:
        - containerPort: 9200
          name: http
          protocol: TCP
        - containerPort: 9300
          name: transport
          protocol: TCP
        resources:
          requests:
            cpu: 10m
            memory: 1Gi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /data/data
          name: data
        - mountPath: /data/logs
          name: log
      dnsPolicy: ClusterFirst
      initContainers:
      - command:
        - sysctl
        - -w
        - vm.max_map_count=262144
        image: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/docker-elasticsearch:0.0.0.12
        imagePullPolicy: Always
        name: init-pod
        resources: {}
        securityContext:
          capabilities:
            add:
            - IPC_LOCK
            - SYS_RESOURCE
          privileged: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: data-docker-elasticsearch
      - name: log
        persistentVolumeClaim:
          claimName: log-docker-elasticsearch
  updateStrategy:
    type: OnDelete

$ kubectl exec docker-elasticsearch-0 -it bash
Error from server (Forbidden): pods "docker-elasticsearch-0" is forbidden: cannot exec into or attach to a privileged container

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): 1.10.2
• Cloud provider or hardware configuration: Azure
• OS (e.g. from /etc/os-release): ubuntu 16.04
• Install tools: acs-engine

[php-coder]

Error from server (Forbidden): pods "docker-elasticsearch-0" is forbidden: cannot exec into or attach to a privileged container

Are you sure that "docker-elasticsearch-0" is the name of your pod? Your containers should have names like pod-0 as far I can see.

[sylr]

Are you sure that "docker-elasticsearch-0" is the name of your pod? Your containers should have names like pod-0 as far I can see.

Yes that's the name of the pod (This pod is part of a StatefulSet instead of a Deployment hence the name) and there is only one container in it besides the initContainer.

[kichristensen]

I have the same issue I'll post the details when I get back to my computer. In this case it is the istio sidecar that uses a privileged init container, a pod without the sidecar doesn't have this issue

[kichristensen]

Here is the details.

I'm running Kubernetes 1.10.3.

My deployment looks like this:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
  labels:
    app: echo
  name: echo-deployment
  namespace: default
spec:
  selector:
    matchLabels:
      app: echo
  template:
    metadata:
      labels:
        app: echo
    spec:
      containers:
      - image: XXX/echo:2.0
        imagePullPolicy: Always
        name: echo-container
        ports:
        - containerPort: 80
          protocol: TCP
      dnsPolicy: ClusterFirst
      imagePullSecrets:
      - name: simcorp-registry
      nodeSelector:
        beta.kubernetes.io/os: linux
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30

Because of the automatic Istio sidecar injection the the resulting pod, looks like this, as you can see the init container is privileged:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    sidecar.istio.io/status: '{"version":"55c9e544b52e1d4e45d18a58d0b34ba4b72531e45fb6d1572c77191422556ffc","initContainers":["istio-init"],"containers":["istio-proxy"],"volumes":["istio-envoy","istio-certs"],"imagePullSecrets":null}'
  labels:
    app: echo
  name: echo-deployment-57d5c75958-prtsp
  namespace: default
spec:
  containers:
  - image: XXXX/echo:2.0
    imagePullPolicy: Always
    name: echo-container
    ports:
    - containerPort: 80
      protocol: TCP
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-8x5xs
      readOnly: true
  - args:
    - proxy
    - sidecar
    - --configPath
    - /etc/istio/proxy
    - --binaryPath
    - /usr/local/bin/envoy
    - --serviceCluster
    - echo
    - --drainDuration
    - 45s
    - --parentShutdownDuration
    - 1m0s
    - --discoveryAddress
    - istio-pilot.istio-system:15007
    - --discoveryRefreshDelay
    - 10s
    - --zipkinAddress
    - zipkin.telemetry:9411
    - --connectTimeout
    - 10s
    - --statsdUdpAddress
    - istio-statsd-prom-bridge.istio-system:9125
    - --proxyAdminPort
    - "15000"
    - --controlPlaneAuthPolicy
    - NONE
    env:
    - name: POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: POD_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: INSTANCE_IP
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIP
    - name: ISTIO_META_POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: ISTIO_META_INTERCEPTION_MODE
      value: REDIRECT
    image: docker.io/istio/proxyv2:0.8.0
    imagePullPolicy: IfNotPresent
    name: istio-proxy
    resources:
      requests:
        cpu: 100m
        memory: 128Mi
    securityContext:
      privileged: false
      readOnlyRootFilesystem: true
      runAsUser: 1337
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /etc/istio/proxy
      name: istio-envoy
    - mountPath: /etc/certs/
      name: istio-certs
      readOnly: true
  dnsPolicy: ClusterFirst
  imagePullSecrets:
  - name: simcorp-registry
  initContainers:
  - args:
    - -p
    - "15001"
    - -u
    - "1337"
    - -m
    - REDIRECT
    - -i
    - '*'
    - -x
    - ""
    - -b
    - 80,
    - -d
    - ""
    image: docker.io/istio/proxy_init:0.8.0
    imagePullPolicy: IfNotPresent
    name: istio-init
    resources: {}
    securityContext:
      capabilities:
        add:
        - NET_ADMIN
      privileged: true
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
  nodeName: k8s-linuxpool-30369698-2
  nodeSelector:
    beta.kubernetes.io/os: linux
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  volumes:
  - name: default-token-8x5xs
    secret:
      defaultMode: 420
      secretName: default-token-8x5xs
  - emptyDir:
      medium: Memory
    name: istio-envoy
  - name: istio-certs
    secret:
      defaultMode: 420
      optional: true
      secretName: istio.default

When I try to exec into the pod:

> kubectl exec -it echo-deployment-57d5c75958-prtsp /bin/sh
Defaulting container name to echo-container.
Use 'kubectl describe pod/echo-deployment-57d5c75958-prtsp -n default' to see all of the containers in this pod.
Error from server (Forbidden): pods "echo-deployment-57d5c75958-prtsp" is forbidden: cannot exec into or attach to a privileged container

Also if I try to specify the specific container:

> kubectl exec -it echo-deployment-57d5c75958-prtsp -c echo-container /bin/sh
Error from server (Forbidden): pods "echo-deployment-57d5c75958-prtsp" is forbidden: cannot exec into or attach to a privileged container

[php-coder]

/sig auth

@kubernetes/sig-auth-bugs

[liggitt]

@tallclair do you know if anything set up by an init container can bleed into something the main containers have access to? things mounted into shared volumes, setcap changes on executables, etc

[IvanovOleg]

I have the same issue with kubernetes v1.11.1 when I try to increase the system vm.max_map_count value for cassandra using the init container. I also tried to set privileged: false in the security context of the cassandra container, but it doesn't override the value set in the init container definition.

[scottlindner]

It appears this is a common issue. I'm having the same issue with the securityContext from the initContainer being applied to the container itself. Here are the tests I performed:

• "initContainer:securityContext:privileged: true" and no container:securityContext, the pod deploys but I cannot kubectl exec into the pod due to the pod being privileged.
• "initContainer:securityContext:privileged: false" and no container:securityContext, the pod goes into an endless crashloop with no meaningful error information provided.
• "initContainer:securityContext:privileged: false" and "initContainer:securityContext:privileged: false", the pod goes into an endless crashloop with no meaningful error information provided.
• no initContainer:securityContext and no container:securityContext, the pod goes into an endless crashloop with no meaningful error information provided.
• completely remove the initContainer and no container:securityContext, the pod deploys and I can kubectl exec into the pod

It seems the issue isn't that the securityContext is inherited from the initContainer, but rather the presence of the initContainer imposes some securityContext default that cannot be changed.

[tallclair]

This is closely related to #55435

Specifically, what is happening is that the DenyEscalatingExec plugin doesn't look at the specific container being exec'd to, but instead looks at all containers in the pod:

kubernetes/plugin/pkg/admission/exec/admission.go

Lines 132 to 151 in 03df9aa

	// isPrivileged will return true a pod has any privileged containers
	func isPrivileged(pod *api.Pod) bool {
	for _, c := range pod.Spec.InitContainers {
	if c.SecurityContext == nil || c.SecurityContext.Privileged == nil {
	continue
	}
	if *c.SecurityContext.Privileged {
	return true
	}
	}
	for _, c := range pod.Spec.Containers {
	if c.SecurityContext == nil || c.SecurityContext.Privileged == nil {
	continue
	}
	if *c.SecurityContext.Privileged {
	return true
	}
	}
	return false
	}

I could see an argument for getting the specific container from the ExecRequest, and only checking the privileged status of that container. OTOH, in a lot of cases we consider the pod to be the security boundary, so you may not want to allow execing into an unpriviged sidecar of a privileged workload, as it could lead to a privilege escalation depending on how much they share.

That said, I don't think the DenyEscalatingExec plugin is very well maintained, and there are some issues with the way it's policy is expressed (or the lack there of). I'm working on a proposal to overhaul PodSecurityPolicy, and I'll try to encorporate this use-case.

[dims]

related? ... #65408 (DenyEscalatingExec: this is kinda dumb)

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[liggitt]

since we do not know what the initContainer did while privileged, we cannot safely assume that nothing done with privileged abilities was made available to the normal containers in the pod.

Since the DenyEscalatingExec admission plugin is optional (and not on by default), I don't anticipate relaxing this restriction. If you want to apply different exec policies, a validating admission webhook can be used instead.

/close

[k8s-ci-robot]

@liggitt: Closing this issue.

In response to this:

since we do not know what the initContainer did while privileged, we cannot safely assume that nothing done with privileged abilities was made available to the normal containers in the pod.

Since the DenyEscalatingExec admission plugin is optional (and not on by default), I don't anticipate relaxing this restriction. If you want to apply different exec policies, a validating admission webhook can be used instead.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.