Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v1.10 AlwaysPullImages admission control order breaks MutatingAdmissionWebhook functionality like Istio #64333

Closed
billpratt opened this issue May 25, 2018 · 31 comments

Comments

@billpratt
Copy link

commented May 25, 2018

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug

/kind feature

What happened:

In v1.10, 7c5f9e0 introduced the ability to not worry about admission control order because it's handled here

AlwaysPullImages is before MutatingAdmissionWebhook. When trying to use Istio sidecar injection, the pod fails to initialize stating

Error creating: pods "sleep-86f6b99f94-qxvq6" is forbidden: spec.initContainers[0].imagePullPolicy: Unsupported value: "IfNotPresent": supported values: "Always"

In v1.9, everything works as expected when placing AlwaysPullImages after MutatingAdmissionWebhook. If you put AlwaysPullImages before MutatingAdmissionWebhook, the same error above occurs.

What you expected to happen:

In v1.10, when AlwaysPullImages and MutatingAdmissionWebhook are turned on, sidecar injection like Istio should work.

How to reproduce it (as minimally and precisely as possible):

  • In v1.10, enable AlwaysPullImages and MutatingAdmissionWebhook admission controllers.
  • Install latest Istio
  • Enable sidecar injection
  • Enabled istio injection on a namespace ie kubectl label namespace default istio-injection=enabled
  • Deploy anything in that namespace
  • Run kubectl describe rs [REPLICA_SET_NAME]. You should see error events similar to Error creating: pods "sleep-86f6b99f94-qxvq6" is forbidden: spec.initContainers[0].imagePullPolicy: Unsupported value: "IfNotPresent": supported values: "Always"
  • Turning off AlwaysPullImages seems to fix Istio

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version): v1.10.3
  • Cloud provider or hardware configuration: acs-engine
  • OS (e.g. from /etc/os-release):
  • Kernel (e.g. uname -a):
  • Install tools:
  • Others:
@billpratt

This comment has been minimized.

Copy link
Author

commented May 25, 2018

@kubernetes/sig-api-machinery-bugs

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

commented May 25, 2018

@billpratt: Reiterating the mentions to trigger a notification:
@kubernetes/sig-api-machinery-bugs

In response to this:

@kubernetes/sig-api-machinery-bugs

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@dims

This comment has been minimized.

Copy link
Member

commented May 25, 2018

looks like this is the istio bug where this came from - istio/istio#5810

@dims

This comment has been minimized.

Copy link
Member

commented May 25, 2018

cc @sttts @hzxuzhonghu thoughts please

@hzxuzhonghu

This comment has been minimized.

Copy link
Member

commented May 26, 2018

I am looking into it.

@dims

This comment has been minimized.

Copy link
Member

commented May 26, 2018

/priority important-soon
/milestone v1.11

@dims

This comment has been minimized.

Copy link
Member

commented May 26, 2018

@k8s-mirror-api-machinery-bugs please feel free to change the priority and/or the milestone

@hzxuzhonghu

This comment has been minimized.

Copy link
Member

commented May 26, 2018

I have a doubt: why enable AlwaysPullImages while inject Istio sidecar with 'IfNotPresent' image pull policy?

@hzxuzhonghu

This comment has been minimized.

Copy link
Member

commented May 26, 2018

That make no sense.

@hzxuzhonghu

This comment has been minimized.

Copy link
Member

commented May 26, 2018

@billpratt

This comment has been minimized.

Copy link
Author

commented May 26, 2018

Would putting AlwaysPullImages after mutating webhook fix the issue? In 1.9, it comes after. See https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#is-there-a-recommended-set-of-admission-controllers-to-use

@sttts

This comment has been minimized.

Copy link
Contributor

commented May 29, 2018

Mutating webhooks are run at the end for the reason to be able to tweak everything other stages of the system have done. We cannot arbitrarily move some plugin adhoc somewhere else. This also breaks compatibility for people relying on the old order.

I can see though that some plugins make more sense at the very end. I wonder whether we need some very limited ordering feature (please not priorities) in the webhook API, maybe a binary flag.

@deads2k

This comment has been minimized.

Copy link
Contributor

commented May 29, 2018

Mutating webhooks are run at the end for the reason to be able to tweak everything other stages of the system have done. We cannot arbitrarily move some plugin adhoc somewhere else.

Correct.

I can see though that some plugins make more sense at the very end.

Everyone wants to be at the very end. Istio is likely to run afoul of PSP as well and changing an ordering for PSP will eliminate some power for mutating admission plugins in their ability to reconfigure objects.

It seems more like istio may need to understand the world it's playing in than trying to twiddle the world around istio (of which there are likely to be many and various use-cases).

@deads2k

This comment has been minimized.

@lavalamp

This comment has been minimized.

Copy link
Member

commented May 29, 2018

Webhooks run after built-in plugins. There is no way to determine the relative ordering. Eventually, we will have to solve that, I think by giving built-ins their own configuration objects (and possibly even making them webhooks).

(Clarification: @sttts and @deads2k did not mean to say that mutating webhooks run after validating webhooks: that's not true, validating webhooks run last.)

The general rule @bgrant0607 has been arguing for is that webhooks should not modify user intent. "User intent" == fields that are already set. There are (at least) two problems with this: you can't tell the difference between a default and user intent. And existing admission plugins (like AlwaysPull) flagrantly violate this rule. But I do think overall life would be better if we could make this the rule.

In this specific case, clearly you cannot enable both AlwaysPull and the Istio sidecar at the same time, no matter what order they are going to be executed in. They have contradictory desires. IMO AlwaysPull is a pretty bad plugin; it should be removed. People who want that behavior can make a webhook enforcing it, which can be more intelligent (e.g., make an exception for images known to be loaded on the node, like Istio's side car).

@deads2k

This comment has been minimized.

Copy link
Contributor

commented May 30, 2018

@liggitt had some thoughts about a two pass approach that he was going to share.

In this specific case, clearly you cannot enable both AlwaysPull and the Istio sidecar at the same time, no matter what order they are going to be executed in.

It may be the case (and I don't know for sure), that the istio plugin didn't express a desire and got a default. However, alwaypullimages is just one case of many. Something like PSP is more likely to be problematic.

@lavalamp

This comment has been minimized.

Copy link
Member

commented May 30, 2018

@k8s-github-robot

This comment has been minimized.

Copy link
Contributor

commented May 31, 2018

[MILESTONENOTIFIER] Milestone Issue Needs Approval

@billpratt @kubernetes/sig-api-machinery-misc

Action required: This issue must have the status/approved-for-milestone label applied by a SIG maintainer. If the label is not applied within 2 days, the issue will be moved out of the v1.11 milestone.

Issue Labels
  • sig/api-machinery: Issue will be escalated to these SIGs if needed.
  • priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts.
  • kind/bug: Fixes a bug discovered during the current release.
Help
@fedebongio

This comment has been minimized.

Copy link
Contributor

commented May 31, 2018

@liggitt

This comment has been minimized.

Copy link
Member

commented Jun 4, 2018

In this specific case, clearly you cannot enable both AlwaysPull and the Istio sidecar at the same time, no matter what order they are going to be executed in

I also had a hard time understanding if Istio cared about the pull policy of the container it injects, or if it was just trying to mimic the server-side defaulting behavior.

  • If Istio does care and requires PullIfNotPresent, then yes, it cannot coexist with AlwaysPull.
  • If it doesn't care, then this is a specific case of two components having opinions about different fields in the same part of an object, which seems useful/important to solve.

IMO AlwaysPull is a pretty bad plugin; it should be removed.

It is currently the only way to ensure imagePullSecrets at the pod level are limited to providing access to images for that pod. A node/container-runtime-level alternative would be welcome, but is not forthcoming.

In any case, assuming Istio doesn't actually care about the pull policy, this issue isn't specific to AlwaysPull. An identical scenario would be encountered with an admission plugin that set or required runAsUser to be a specific value on all containers in the pod.

Solutions that come to mind:

  • Manual ordering of plugins relative to each other. This is very likely to result in partial orders, and requires plugins be extremely granular (even to the point of breaking up a single coherent plugin that touches a couple parts of an object, just so those individual operations can be painstakingly placed in the global order)
  • Automatic ordering of plugins based on the parts of the object they touch. For example, admission that adds pod containers would get run before plugins that modify subfields of containers. This doesn't seem feasible to express, and runs into the same granularity issues as manual ordering.
  • Multiple mutating admission passes. In the case where a mutating admission plugin makes structural additions (e.g. adding a new container), we could rerun the mutating phase to give all plugins visibility to the new substruct, and the opportunity to enforce/default/reject fields on that substruct.
@mlbiam

This comment has been minimized.

Copy link

commented Aug 7, 2018

Multiple mutating admission passes. In the case where a mutating admission plugin makes structural additions (e.g. adding a new container), we could rerun the mutating phase to give all plugins visibility to the new substruct, and the opportunity to enforce/default/reject fields on that substruct.

to address #67050 i would support this. as a webhook dev, i want to do the minimum possible and inherit the controls the system puts in place. In my own request I don't want to have to re-implement the logic from previous admission plugins in my webhook. To this end, I want my mutating webhook to run first so as to be as close to the user's initial request as possible. This way I can inject my changes (update securityContext, add containers, volumes, etc) without having to know what the name of a service account's token is. This strategy leads to:

  1. Much more readable code
  2. More secure webhooks
  3. Better reuse of webhook code

As an example, in order to inject the kerberos sidecar in our webhook we had to generate the following patches in OPA:

krb_sidecar_patch = [
    {
        "op": "replace",
        "path": "/spec/serviceAccountName",
        "value": "run-as-user"
    },
    {
        "op": "replace",
        "path": "/spec/imagePullSecrets/0/name",
        "value": user_data["sa-dockercfg"]
    },
    {
        "op": "replace",
        "path": "/spec/volumes/0/name",
        "value": user_data["sa-token"]
    },
    {
        "op": "replace",
        "path": "/spec/volumes/0/secret/secretName",
        "value": user_data["sa-token"]
    },
    {
        "op": "replace",
        "path": "/metadata/annotations/openshift.io~1scc",
        "value": sprintf("run-as-%s",[kerb_sidecar_data.uid])
    },
    {
        "op": "add",
        "path": "/spec/securityContext/runAsUser",
        "value": kerb_sidecar_data.uidnumber
    },
    {
        "op": "replace",
        "path": "/spec/containers/0/securityContext/runAsUser",
        "value": kerb_sidecar_data.uidnumber
    },
    {
        "op": "replace",
        "path": "/spec/containers/0/volumeMounts/0/name",
        "value": user_data["sa-token"]
    },
    {
        "op": "add",
        "path": "/spec/containers/1",
        "value": {
            "resources": null,
            "image": "mlbiam/krbsidecar",
            "name": "sidecar",
            "env": [
                {
                    "name": "USER_PRINCIPAL",
                    "value": kerb_sidecar_data.userPrincipalName
                },
                {
                    "name": "KRB5_CONFIG",
                    "value": "/etc/krb-conf/krb5.conf"
                }
            ],
            "volumeMounts": [
                {
                    "name": "krb-kt",
                    "mountPath": "/krb5"
                },
                {
                    "name": "krb-conf",
                    "mountPath": "/etc/krb-conf"
                },
                {
                    "name": "dshm",
                    "mountPath": "/dev/shm"
                }
            ],
            "imagePullPolicy": "Always",
            "terminationMessagePolicy": "File",
            "terminationMessagePath": "/dev/termination-log"
        }
    },
    {
        "op": "add",
        "path": "/spec/containers/0/env",
        "value": [
            {
                "name": "KRB5_CONFIG",
                "value": "/etc/krb-conf/krb5.conf"
            }
        ]
    },
    {
        "op": "add",
        "path": "/spec/containers/0/volumeMounts/1",
        "value": {
            "name": "krb-conf",
            "mountPath": "/etc/krb-conf"
        }
    },
    {
        "op": "add",
        "path": "/spec/containers/0/volumeMounts/2",
        "value": {
            "name": "dshm",
            "mountPath": "/dev/shm"
        }
    },
    {
        "op": "add",
        "path": "/spec/volumes/1",
        "value": {
            "name": "krb-kt",
            "hostPath": {
                "path": sprintf("/home/%s/%s/keytab",[kerb_sidecar_data.domain,kerb_sidecar_data.uid]),
                "type": "Directory"
            }
        }
    },
    {
        "op": "add",
        "path": "/spec/volumes/2",
        "value": {
            "name": "krb-conf",
            "configMap": {
                "name": "krb5-config"
            }
        }
    },
    {
        "op": "add",
        "path": "/spec/volumes/3",
        "value": {
            "name": "dshm",
            "emptyDir": {
                "medium": "Memory"
            }
        }
    },
    {
        "op": "replace",
        "path": "/spec/serviceAccount",
        "value": "run-as-user"
    },
    {
        "op": "add",
        "path": "/metadata/annotations/com.tremolosecurity.openshift.krb5_sidecar.added",
        "value": "krb5_sidecar"
    }
]

where as if the webhook could be run at the beginning of the chain my code would be:

krb_sidecar_patch = [
    {
        "op": "replace",
        "path": "/spec/serviceAccountName",
        "value": "run-as-user"
    },
    
    {
        "op": "add",
        "path": "/spec/securityContext/runAsUser",
        "value": kerb_sidecar_data.uidnumber
    },
    
    {
        "op": "add",
        "path": "/spec/containers/1",
        "value": {
            "resources": null,
            "image": "mlbiam/krbsidecar",
            "name": "sidecar",
            "env": [
                {
                    "name": "USER_PRINCIPAL",
                    "value": kerb_sidecar_data.userPrincipalName
                },
                {
                    "name": "KRB5_CONFIG",
                    "value": "/etc/krb-conf/krb5.conf"
                }
            ],
            "volumeMounts": [
                {
                    "name": "krb-kt",
                    "mountPath": "/krb5"
                },
                {
                    "name": "krb-conf",
                    "mountPath": "/etc/krb-conf"
                },
                {
                    "name": "dshm",
                    "mountPath": "/dev/shm"
                }
            ],
            "imagePullPolicy": "Always",
            "terminationMessagePolicy": "File",
            "terminationMessagePath": "/dev/termination-log"
        }
    },
    {
        "op": "add",
        "path": "/spec/containers/0/env",
        "value": [
            {
                "name": "KRB5_CONFIG",
                "value": "/etc/krb-conf/krb5.conf"
            }
        ]
    },
    {
        "op": "add",
        "path": "/spec/containers/0/volumeMounts/1",
        "value": {
            "name": "krb-conf",
            "mountPath": "/etc/krb-conf"
        }
    },
    {
        "op": "add",
        "path": "/spec/containers/0/volumeMounts/2",
        "value": {
            "name": "dshm",
            "mountPath": "/dev/shm"
        }
    },
    {
        "op": "add",
        "path": "/spec/volumes/1",
        "value": {
            "name": "krb-kt",
            "hostPath": {
                "path": sprintf("/home/%s/%s/keytab",[kerb_sidecar_data.domain,kerb_sidecar_data.uid]),
                "type": "Directory"
            }
        }
    },
    {
        "op": "add",
        "path": "/spec/volumes/2",
        "value": {
            "name": "krb-conf",
            "configMap": {
                "name": "krb5-config"
            }
        }
    },
    {
        "op": "add",
        "path": "/spec/volumes/3",
        "value": {
            "name": "dshm",
            "emptyDir": {
                "medium": "Memory"
            }
        }
    },
    {
        "op": "add",
        "path": "/metadata/annotations/com.tremolosecurity.openshift.krb5_sidecar.added",
        "value": "krb5_sidecar"
    }
]

The second block of code eliminates updates to individual container's service accounts and is portable between k8s distros (since its no longer coded to openshift's annotations). It also eliminates the need for me to lookup secret names for specific service accounts leading to fewer possible chances to leak information.

@lavalamp

This comment has been minimized.

Copy link
Member

commented Oct 22, 2018

Followup to my comment above (#64333 (comment)):

As part of apply, we've developed the concept of "field sets"; we will be using this to track which fields the user deliberately set (and implicitly, which were defaulted). We probably will use this to indicate in the object which fields each mutating webhook set--we haven't written that part yet.

Next we have a choice:

  1. We could reject a webhook's proposed patch if it modifies a field set by the user (or prior webhook?).
  2. We could accept the patch and give the webhook ownership of the conflicting field.
  3. We could accept the patch and leave the conflicting field owned by the user.

(1) doesn't solve things, it just gives a slightly better error message ("webhook X and Y conflict; talk to your cluster administrator")
(2) means the next time user applies, they'll be warned about the conflict and will have to either modify their config or "force" to override (assuming the field is mutable). This is annoying if the user is a CI/CD system that doesn't force without intervention--we'd prefer to have failed the first time rather than check in a change which then can't re-applied.
(3) means the user doesn't have to force, but will silently override the field's value (if it's mutable).

One example that came up in the discussion was a hypothetical webhook that took an image name:tag and resolved it to a hash, and put that in the image field (of e.g. a deployment's pod template), and didn't re-resolve on an update unless the user changes the image name:tag. Since on an update, webhooks will see the object post-apply, only (3) above allows the user to make a change without forcing every time. I'm not sure how important this use case is.

Anyway if folks have use cases for choices 1 or 2 above, now would be a great time to give us that feedback.

cc @apelisse @jennybuckley

@fejta-bot

This comment has been minimized.

Copy link

commented Jan 20, 2019

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@faheem-cliqz

This comment has been minimized.

Copy link

commented Jan 20, 2019

/remove-lifecycle stale

@deads2k

This comment has been minimized.

Copy link
Contributor

commented Jan 21, 2019

@mbohlool @liggitt @kubernetes/sig-api-machinery-misc I see this as an area that needs to be resolved before admission webhooks can move to GA.

@runyontr

This comment has been minimized.

Copy link
Contributor

commented Jan 24, 2019

To me, it seems as though there are two types of mutating WebHooks:

  1. Changing functionality (e.g. Istio adding a sidecar)
  2. Conformance (e.g. AlwaysPullImage)

The Conformance WebHooks are typically tied to a Validation WebHook that then validates the object adhere's to policy.

Could the structure of validation AdmissionsWebhooks be modified to contain a reference to a "remediation" MutatingWebHook that that gets executed on failure, and then revalidated. If it still fails, then the object is rejected. This allows for real mutation hooks to be executed before validation (as is currently implemented) but differentiation MHW that only serve to ensure functionality of a Validation WebHook.

Another idea is as follows:
When a Mutating WebHook changes an object, consider it a new submission and require it to go back through all webhooks. This requires an assumption that MWH are idempotent.

Concerns: Would require logic to ensure that objects aren't getting caught in a modification loop with a bad acting/conflicting webhooks.

Pros: It would ensure that every MutatingWebhook has been applied to the object that gets passed to the validation webhooks.

@hughsaunders

This comment has been minimized.

Copy link

commented Mar 16, 2019

I came across this issue when attempting to use the k8s api from a container injected via a mutating webhook. It failed because the service account token volume mount doesn't exist. I think the reason for this is that the service account admission controller runs before the MAWH.

@runyontr mentioned the idea of dividing MAWH into two categories, how about early and late? Early MAWHs run early and get access to the request as posted by the user. Late MAWHs run late, and see the results of most of the other plugins.

For my usecase, I would use an early MAWH to add another container into the podspec, this would then go through the serviceaccount plugin where both containers would have their service account token volume mounts added.

hughsaunders referenced this issue in cyberark/sidecar-injector Mar 16, 2019

Adds Service Account Token Volume Mount
This commit copies the service account token volume vmount
from an existing container in the pod to the injected pod.
This ensures that injected containers can access the k8s api.
This is a fragile hack and should be fixed somewhere else(!)
@runyontr

This comment has been minimized.

Copy link
Contributor

commented Apr 3, 2019

It seems there are other tools that solve this problem, and we might be able to leverage their implementations

Systemd

https://www.freedesktop.org/software/systemd/man/systemd.unit.html
Add two fields to the WebHook definitions and reference objects that it should run before/after

Before []corev1.ObjectReference
After   []corev1.ObjectReference

chkconfig

https://linux.die.net/man/8/chkconfig

RunLevel might be already captured with the two types of WebHooks (Mutating and Validation) but we may want to split them out and have different RunLevels for each type. Within a type, we could then specify the run position

# What RunLevel to be in.  Valid 0-6
Level  int
# What Position within the RunLevel to be in. Valid >0
Position int
@lavalamp

This comment has been minimized.

Copy link
Member

commented Apr 3, 2019

We haven't thought of a good solution yet, where "good" is some combination of:

  • solves the problem
  • doesn't make the system insanely hard to configure or understand
    • in particular, it mustn't make it impossible to share webhook configurations, i.e., to other people / clusters
  • doesn't make the system insanely non-performant

Requiring/permitting a graph (as in systemd) would:

  • make the system hard to understand (imagine debugging your webhook flow in this system)
  • make it difficult or impossible to share a webhook configuration (it might now reference a bunch of webhooks that aren't installed, and/or fail to reference some which are).
  • doesn't solve webhooks which make transitive changes (A will change B will change C will change A)

"Run levels" are easier to understand/explain/predict but still have the latter two problems.

Our current working plan is basically to run every mutating webhook a second time, if a webhook that ran after it changed something. (max 2 runs)

There may be some factors I've forgotten to mention; @liggitt and @mbohlool have convinced me it's a hard problem.

Note that today you can explicitly control the order of webhooks by changing the webhook configuration name and/or combining them into the same configuration object.

@liggitt

This comment has been minimized.

Copy link
Member

commented Jun 10, 2019

fixed by #78080 in 1.15

@liggitt liggitt closed this Jun 10, 2019

@runyontr runyontr referenced this issue Jul 17, 2019

Closed

REQUEST: New membership for runyontr #1014

6 of 6 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.