Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Ensuring only images are from approved sources are run #22888
A customer has asked for a feature which would ensure that only certain "approved" binaries are run on the cluster. This issue is to get feedback from other users on whether this would be useful, and to clarify requirements, and to see if some simple implementations would be sufficient.
The threat some users are trying to protect against is that an otherwise authorized user of a kubernetes cluster accidentally or intentionally creates a pod that runs a binary that was built from untrusted, possibly malicious, source code. It might actually cause direct harm, or just require costly auditing after the fact. Auditing requirements may be dictated by Service Organization Controls.
Policy enforcement in the apiserver is easier to change and easier to surface errors, but probably more work to get something we all agree on working. Policy enforcement or docker can apply to http and file source pods, and kubelet or docker are in better position to talk to a private registry, if more complex interaction is later needed. A docker-based solution could be applicable to all users including non-kubernetes users. A kubelet or apiserver solution would apply to multiple container runtimes. Only the apiserver can easily have a per-user policy, since only it is aware what user is creating the object. Only apiserver or kubelet can have a per-namespace policy, since docker does not know or care what a namespace is.
Docker Notary and Docker Content Trust
Docker Content Trust checks the integrity of the image (e.g. to ensure it was not modified by a man-in-the-middle), and only allows images with signed tags to run.
Docker Content Trust uses Docker Notary.
Docker Notary is a general solution to compute checksums of collections of files, and publish signed checksums of those files. Docker Notary could also sign things not in images, like configMaps, for example.
The docker client (CLI, or the go language docker client library, in the case of Kubelet) is responsible for setting whether trust should be enabled, and what notary server to use, on each call to docker.
Docker content trust only applies to image tags. A user who asks for a specific content hash, e.g.
So, it isn't clear how to use Docker Content Trust by itself as a solution to prevent untrusted images from being run.
rkt trust works similarly to Docker Content Trust (and predates it.) It controls which registries are trusted.
Integrity Measurement Architecture
The Integrity Measurement Architecture (IMA) provides a way for parties to remotely verify the integrity of programs running on a machine.
It appears to be a quite complex system. It requires a kernel that supports it, and I imagine it is complex to debug. I am not considering it at this time.
Openshift Builds and Imagestreams
Openshift provides an image build system and a private registry for each cluster, or each tenant of a cluster. Having image builds happen on the same cluster as images run may be convenient in some cases, but do we need to mandate this to use this feature? Some users have their own build system already setup
Kubernetes has an open issue to provide a per-cluster registry as an add-on (#1319).
Re implementation, why not mixing docker content trust with existing kubernetes authz approach.
That is, apiservers will perform the image tag validation (without hash), while docker content trust will verify that the image is from a trusted authority.
This can be achieved by forcing content validation on all image pulls (for docker, either enable content trust in the daemon level or use the authz plugin).
We would benefit from and use this functionality.
However, it would be valuable only if the whitelisting applies to all of the containers that are part of the dependency chain for a given container.
E.g., if we have an organizational policy that we build all base containers in-house instead of using anything from docker hub, we would need to ensure that when we run the "fooService" container that the FROM for that container is an allowed container; and that that container's FROM line is allowable, and so on until there is no FROM line.
Here's the approach we were going to take in OpenShift (where we have integrated the registry into the platform), allowing the following sets of rules:
(in the rules below, "use" means "can be the input to a pod run, a build, or be tagged into the system for reuse")
We're still trying to gather additional use cases, but most of them here are at the platform level, not at the node level. I think it's assumed that if node level verification can be performed, it should, but that most administrative policy would happen above that level.
Operational teams may delegate access to build images to users, but still
On Tue, Mar 22, 2016 at 8:15 PM, Eric Tune email@example.com wrote:
Are you planning to have an "Image Policy" object that is stored in the apiserver?
You said in item 7 above that you would Perform a call to a remote system prior to allowing an image to be run. Are you guys envisioning an admission control plugin in apiserver that calls out to an "image approver" service? Or a kubelet or docker call to an image approval system? Or some other mechanism?
For item 7, what info goes to the remote system? Image name? What else?
Do you have a use case for overriding the policy in an emergency? Would an admin role be able to bypass these checks? If so, how is username or role propagated to the point of enforcement?
@erictune belatedly (sorry about that); we do a lot of automated builds with Jenkins, though the security story with Jenkins isn't great. The gist of your suggestion is that it'd definitely be easier on Kubernetes if it needed only to be a link in a trust chain- i.e., trust a registry that trusts a build pipeline, etc.; and there's definitely validity there from the k8s engineering perspective.
In our case (which I suspect is reflective of the situation of the long tail of K8S users), establishing trust throughout the entire build process is difficult, whereas Kubernetes represents a place where it can be done once, and, if done right, be done effectively.
If I were to have built a trusted pipeline, it would be largely wasted effort since Kubernetes could still be directed to run an image originating from elsewhere. Knowing that, I would heretofore have no reason to embark on such work.
Whereas if Kubernetes were the hypothetically only place I could build trust, it would be effective. If Kubernetes will only run images that pass some authorizer, then it doesn't matter where the candidate images come from; letting me secure my registry and build pipeline as an exercise in defense-in-depth.)
tl;dr: Our pipeline isn't trusted, and I appreciate the benefits of Kubernetes being just a link in a trust chain, but I think implementing the security more directly in K8S would be effective and powerful.
Summarizing conversation so far in my own words:
Questions I still have:
And OpenShift plans on implementing the remote hook for this at some point
On Wed, Aug 10, 2016 at 4:30 PM, Q-Lee firstname.lastname@example.org wrote:
added a commit
Aug 23, 2016
We have been exploring enabling Docker Image Signing in our clusters and we looked into the Image Provenance Architecture (https://github.com/kubernetes/community/blob/master/contributors/design-proposals/image-provenance.md) in the process. We believe that there is a use-case in which we can augment the architechture for two additional features.
The two additional "features" we have identified are:
Image Tag Rewriting
This feature comes about from an issue highlighted in the architecture documents as follows;
By performing image rewriting, for example, the image
This concern stems from the scenario where an attacker has compromised a registry server. In this case, we believe it is feasible for the attacker (via fingerprinting), to determine a request from a webhook verifier from a kubelet pulling the image. In that case, the malicious attacker is able to always serve the correct image to the verifier and the tampered image to the kubelet.
To do this, the webhook would have to return a list of image names to rewrite. i.e. it would have to return a list for
This feature comes about from the possibility that the image that the user wants to use is part of a private repository. Most commonly, auth details are stored in
A preliminary implementation idea would be to initialize the admission controller with an
(I'm not too sure if there is a better way to propagate this information)
W.R.T. Image Tag Rewriting, I know this is being discussed but I'm not sure where it's written down. A decision needs to be made about where the translation point happens, and for consistency reasons that probably needs to be before the image name reaches the kubelet for running.
Please see #31524 for a discussion of ImagePullSecret propagation. There are a lot of concerns with it, and I've convinced myself that something like an oauth flow (that kubernetes doesn't need to know about) can solve the problem. I'd love to hear different ideas if you have them!
Thanks @ecordell for the feedback! By the way, do you have a image policy webhook that I can take a look at It would be great if there is something that I can play around with!
Agreed! It would be nice if somehow I can view/jump in on this discussion. I've played around with admission controller where I did rewriting at the admission controller level, which seems like an OK idea (imo).
I have do delve a bit deeper into the discussion, but it seems like there's a differentiation between use cases of ImagePullSecrets VS Non-ImagePullSecrets and External Party VS Extension of the Cluster.
EDIT: Reading through the ImagePullSecrets thread: I guessed I only looked at the "ImagePullSecrets" case.
I think oauth is a nice framework to do things. I will have to look into closer details. The registries and notary servers already support the oauth framework to perform the delegation, which is nice. :).
Just for sharing: In playing around with this, I had a very simplistic view on this, in terms of just extracting the secrets and passing it along, in the same way that the kubelet would consume the
I have implemented the image rewriting on the admission controller: https://github.com/kubernetes/kubernetes/compare/master...lumjjb:img-webhook-con-rewrite?expand=1
Do you think we should have this be merged? Or should we discuss this more?
This addresses the problem in the documentation (https://github.com/kubernetes/community/blob/master/contributors/design-proposals/image-provenance.md ):
However this is not just a race, since it is possible for a malicious actor to probably fingerprint whether it is the docker agent or webhook that is querying for an image (and provide the tampered response only to the docker agent running on the k8s worker).
I sketched this out a couple of weeks ago with a slightly different approach:
(I didn't flesh out the tests or do the codegen)
I was concerned that anything we do here would be usurped by kubernetes/community#132, which intends to rework admission controllers and explicitly avoid mutating webhook backends for the first version.
This is true, but I think you're just highlighting that any admission controller that can mutate the request (including a webhook backend in this case) needs to be as trusted and as secured as any of the rest of the control plane.
Mutating admission web hooks seems likely in 1.8 - if you can wait that long it's one approach.…
On Tue, Jun 6, 2017 at 8:11 AM, Evan Cordell ***@***.***> wrote: @lumjjb <https://github.com/lumjjb> I sketched this out a couple of weeks ago with a slightly different approach: master...ecordell:imagereviewwebhook-digest <master...ecordell:imagereviewwebhook-digest> (I didn't flesh out the tests or do the codegen) I was concerned that anything we do here would be usurped by kubernetes/community#132 <kubernetes/community#132>, which intends to rework admission controllers and explicitly avoid mutating webhook backends for the first version. However this is not just a race, since it is possible for a malicious actor to probably fingerprint whether it is the docker agent or webhook that is querying for an image (and provide the tampered response only to the docker agent running on the k8s worker). This is true, but I think you're just highlighting that any admission controller that can mutate the request (including a webhook backend in this case) needs to be as trusted and as secured as any of the rest of the control plane. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#22888 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABG_pxQz_f02PPhyOrGO72fgG6Gt2zJeks5sBUHagaJpZM4HvLIL> .
I like your dictionary implementation more actually. Looks a lot neater.
I wrote mine in consideration that there may be more things in the container spec in the future that we may require rewriting, but seems like there isn't a usecase in sight at the moment.
Given the new design, would the image policy admission controller be an
Also, should we update the document to express the intention of the
For the current build i suppose it could be possible to add a flag in the config file to "trust" the webhook to perform rewriting or disallow the rewriting (as you pointed out, this still won't work with the new design immediately if it is an
@smarterclayton It would be nice to be able to get this in a little sooner than that if possible :). Understand that there may be a need to rewrite the code/redesign parts of it later.
Issues go stale after 90d of inactivity.
Prevent issues from auto-closing with an
If this issue is safe to close now please do so with
Send feedback to sig-testing, kubernetes/test-infra and/or
Stale issues rot after 30d of inactivity.
If this issue is safe to close now please do so with
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.