Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sidecar-injector with SDS enabled breaks pod securityContext #20409

Closed
btoonk opened this issue Jan 22, 2020 · 1 comment · Fixed by #20428
Closed

sidecar-injector with SDS enabled breaks pod securityContext #20409

btoonk opened this issue Jan 22, 2020 · 1 comment · Fixed by #20428

Comments

@btoonk
Copy link
Contributor

@btoonk btoonk commented Jan 22, 2020

Bug description

The pod securityContext is patched by the sidecar-injector when sds is enabled.
But it looks like the pod securityContext gets overwritten instead of properly patched.

This will also overwrite pod securityContext fields like runAsUser which results in a POD without the runAsUser pod securityContext field.

See:
https://github.com/istio/istio/blob/master/pkg/kube/inject/inject.go#L848
https://github.com/istio/istio/blob/master/pkg/kube/inject/webhook.go#L576

This only impacts the pod securityContext spec not the securityContext field in container spec.

[ X] Configuration Infrastructure
[ ] Docs
[ ] Installation
[ X] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[ X] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure

Expected behavior

When enabling SDS, we expect a Pod with a pod securityContext spec to be scheduled with these pod securityContext applied and with only the FSGroup added/overwritten.

Steps to reproduce the bug

  • Enable SDS (istioctl manifest apply --set profile=sds)
  • Deploy helloworld:
kind: Deployment
metadata:
  name: helloworld
  namespace: test
spec:
  selector:
    matchLabels:
      app: helloworld
  replicas: 2
  template:
    metadata:
      labels:
        app: helloworld
    spec:
      securityContext:
        runAsUser: 1000
      containers:
      - name: helloworld
        image: docker.io/istio/examples-helloworld-v1
        ports:
        - containerPort: 5000

In our case this will result in the following events because we only allow NonRoot users with psp:
Warning FailedCreate 22s replicaset-controller Error creating: pods "helloworld-78969cf685-p8h8n" is forbidden: unable to validate against any pod security policy: []

If we add the runAsUser in the containers securityContext spec the pods will be scheduled correctly.

kind: Deployment
metadata:
  name: helloworld
  namespace: test
spec:
  selector:
    matchLabels:
      app: helloworld
  replicas: 2
  template:
    metadata:
      labels:
        app: helloworld
    spec:
      securityContext:
        runAsUser: 1000
      containers:
      - name: helloworld
        image: docker.io/istio/examples-helloworld-v1
        ports:
        - containerPort: 5000
        securityContext:
          runAsUser: 1000

In the istio-sidecar-injector log we see the following in the "info AdmissionResponse: patch" message:
{"op":"add","path":"/spec/securityContext","value":{"fsGroup":1337}}

With sds disabled we see:
{"op":"add","path":"/spec/securityContext","value":{"runAsUser":1000,"fsGroup":1}}

Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)

istioctl version --remote
client version: 1.4.3
citadel version: 1.4.3
galley version: 1.4.3
nodeagent version:
nodeagent version:
nodeagent version:
nodeagent version:
nodeagent version:
nodeagent version:
pilot version: 1.4.3
policy version: 1.4.3
sidecar-injector version: 1.4.3
telemetry version: 1.4.3
data plane version: 1.4.3 (4 proxies)

How was Istio installed?

istioctl manifest apply --set profile=sds

Environment where bug was observed (cloud vendor, OS, etc)

We're running on GKE, tested with version v1.13.11-gke.14

@rlenglet

This comment has been minimized.

Copy link
Member

@rlenglet rlenglet commented Jan 22, 2020

The unable to validate against any pod security policy: [] error message indicates that this issue is very similar to #17318. The PSP validation phase fails because the Istio sidecar injection webhook rewrites the pod spec in a way that is inconsistent with the PSP matched by the PSP admission controller.

The Istio webhook is removing all fields from the pod's securityContext (except fsGroup), and that is the reason it's failing PSP validation. The fix is to stop removing any field that is already set either in the original user's spec or by the PSP admission controller, and only set the fsGroup field.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Linked pull requests

Successfully merging a pull request may close this issue.

4 participants
You can’t perform that action at this time.