New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensuring only images are from approved sources are run #22888

Closed
erictune opened this Issue Mar 12, 2016 · 32 comments

Comments

Projects
None yet
@erictune
Member

erictune commented Mar 12, 2016

A customer has asked for a feature which would ensure that only certain "approved" binaries are run on the cluster. This issue is to get feedback from other users on whether this would be useful, and to clarify requirements, and to see if some simple implementations would be sufficient.

Background

The threat some users are trying to protect against is that an otherwise authorized user of a kubernetes cluster accidentally or intentionally creates a pod that runs a binary that was built from untrusted, possibly malicious, source code. It might actually cause direct harm, or just require costly auditing after the fact. Auditing requirements may be dictated by Service Organization Controls.

Requirements Questions

  1. Is it sufficient to whitelist what image repositories are allowed? For example, allow only image names with prefixes gcr.io/myproject/* or docker.io/library/ruby. Or is finer control, such whitelisting individual tags or tag patterns needed? Or even specific binary content hashes.
    1. Docker supports specifying images at a specific checksum, like docker pull someimage@sha256:d149ab53f8718e987c3a3024bb8aa0e2caadf6c0328f1d9d850b2a2a67f2819a. Is this allowed, and if so how would it work with a tag whitelist?
  2. Should the whitelist check be intersected with Docker Content Trust tag signing checks?
  3. Does it need to work for all container runtimes (docker, rkt, etc) or is just Docker enough for now?
  4. Is a single, cluster-wide policy useful, at least as a first-pass. Will we eventually will want policy for each user (to allow the cluster admin to run any image) or per namespace (to ensure each "tenant" in a multi-tenant cluster runs images from its own trusted build chain).
  5. Is a policy that is expresses as a flag useable enough?
  6. Is the policy mandatory, or would after-the-fact auditing be sufficient for most users? If it is mandatory, how easy does it need to be to override it in an emergency (e.g. docker hub is down, and our policy says only pull docker images, and we need to push a new image now due to a critical bug; how do I do that?)
  7. Is image whitelisting useful by itself? Or are restrictions on other parts of a Pod spec (volumes, configMaps, env vars) needed to meet auditing requirements?

Implementation Ideas

  1. docker auth plugin such as twistlock which only allows certain images to run
  2. kubelet flag that contains an image prefix whitelist
  3. admission controller that checks image strings in pods against a policy objects stored in the APIserver.
  4. contributed controller that watches for pods with disallowed images and sends alerts somewhere.

Policy enforcement in the apiserver is easier to change and easier to surface errors, but probably more work to get something we all agree on working. Policy enforcement or docker can apply to http and file source pods, and kubelet or docker are in better position to talk to a private registry, if more complex interaction is later needed. A docker-based solution could be applicable to all users including non-kubernetes users. A kubelet or apiserver solution would apply to multiple container runtimes. Only the apiserver can easily have a per-user policy, since only it is aware what user is creating the object. Only apiserver or kubelet can have a per-namespace policy, since docker does not know or care what a namespace is.

Related Technology

Docker Notary and Docker Content Trust

Docker Content Trust checks the integrity of the image (e.g. to ensure it was not modified by a man-in-the-middle), and only allows images with signed tags to run.

Docker Content Trust uses Docker Notary.

Docker Notary is a general solution to compute checksums of collections of files, and publish signed checksums of those files. Docker Notary could also sign things not in images, like configMaps, for example.

The docker client (CLI, or the go language docker client library, in the case of Kubelet) is responsible for setting whether trust should be enabled, and what notary server to use, on each call to docker.

Docker content trust only applies to image tags. A user who asks for a specific content hash, e.g. docker pull someimage@sha256:d149ab53f8718e987c3a3024bb8aa0e2caadf6c0328f1d9d850b2a2a67f2819a, always succeeds.

So, it isn't clear how to use Docker Content Trust by itself as a solution to prevent untrusted images from being run.

rkt Trust

rkt trust works similarly to Docker Content Trust (and predates it.) It controls which registries are trusted.

Integrity Measurement Architecture

The Integrity Measurement Architecture (IMA) provides a way for parties to remotely verify the integrity of programs running on a machine.
https://sourceforge.net/p/linux-ima/wiki/Home/#integrity-measurement-architecture-ima

It appears to be a quite complex system. It requires a kernel that supports it, and I imagine it is complex to debug. I am not considering it at this time.

Openshift Builds and Imagestreams

Openshift provides an image build system and a private registry for each cluster, or each tenant of a cluster. Having image builds happen on the same cluster as images run may be convenient in some cases, but do we need to mandate this to use this feature? Some users have their own build system already setup

Per-cluster registries

Kubernetes has an open issue to provide a per-cluster registry as an add-on (#1319).
On GKE, clusters have access to a Project-wide Image registry. However, some customers may have existing, private registries. We should not require them to change that to take advantage of this feature.

@erictune

This comment has been minimized.

Show comment
Hide comment
@erictune

erictune Mar 12, 2016

Member

@kubernetes/sig-auth @liron-l @dimastopel @wteiken

Member

erictune commented Mar 12, 2016

@kubernetes/sig-auth @liron-l @dimastopel @wteiken

@smarterclayton

This comment has been minimized.

Show comment
Hide comment
@smarterclayton

smarterclayton Mar 12, 2016

Contributor

We had been planning to implement this soon, I'll follow up here with our approach (not necessarily exactly equal because we have all the image metadata close at hand, but can make sure what can be shared is shared).

Contributor

smarterclayton commented Mar 12, 2016

We had been planning to implement this soon, I'll follow up here with our approach (not necessarily exactly equal because we have all the image metadata close at hand, but can make sure what can be shared is shared).

@liron-l

This comment has been minimized.

Show comment
Hide comment
@liron-l

liron-l Mar 12, 2016

Re implementation, why not mixing docker content trust with existing kubernetes authz approach.

That is, apiservers will perform the image tag validation (without hash), while docker content trust will verify that the image is from a trusted authority.

This can be achieved by forcing content validation on all image pulls (for docker, either enable content trust in the daemon level or use the authz plugin).
Then, extend kubernetes authz approach (which already has user/ns granularity) in apiservers with image whitelisting.

liron-l commented Mar 12, 2016

Re implementation, why not mixing docker content trust with existing kubernetes authz approach.

That is, apiservers will perform the image tag validation (without hash), while docker content trust will verify that the image is from a trusted authority.

This can be achieved by forcing content validation on all image pulls (for docker, either enable content trust in the daemon level or use the authz plugin).
Then, extend kubernetes authz approach (which already has user/ns granularity) in apiservers with image whitelisting.

@pdbogen

This comment has been minimized.

Show comment
Hide comment
@pdbogen

pdbogen Mar 14, 2016

We would benefit from and use this functionality.

However, it would be valuable only if the whitelisting applies to all of the containers that are part of the dependency chain for a given container.

E.g., if we have an organizational policy that we build all base containers in-house instead of using anything from docker hub, we would need to ensure that when we run the "fooService" container that the FROM for that container is an allowed container; and that that container's FROM line is allowable, and so on until there is no FROM line.

pdbogen commented Mar 14, 2016

We would benefit from and use this functionality.

However, it would be valuable only if the whitelisting applies to all of the containers that are part of the dependency chain for a given container.

E.g., if we have an organizational policy that we build all base containers in-house instead of using anything from docker hub, we would need to ensure that when we run the "fooService" container that the FROM for that container is an allowed container; and that that container's FROM line is allowable, and so on until there is no FROM line.

@smarterclayton

This comment has been minimized.

Show comment
Hide comment
@smarterclayton

smarterclayton Mar 14, 2016

Contributor

Here's the approach we were going to take in OpenShift (where we have integrated the registry into the platform), allowing the following sets of rules:

(in the rules below, "use" means "can be the input to a pod run, a build, or be tagged into the system for reuse")

  1. Only use images on the platform that have a set of known "base layers" (as mentioned above), defined by a specific policy
  2. Only use images on the platform from a whitelist/blacklist of known registries
  3. Only allow images to be used that come from the internal registry (so all use requires the import action)
  4. Only allow images to be used that have a certain label or annotation (only settable by admins) - that annotation could be anything, but typical use cases would be an admin scanning process that flags images that have been scanned. Until they're scanned, the platform will refuse to create pods that use those images
  5. Require images to have an attached / detached signature (really just a special case of 4 above) before being used
  6. Limit images that are older than a certain date from being run
  7. Perform a call to a remote system prior to allowing an image to be run

We're still trying to gather additional use cases, but most of them here are at the platform level, not at the node level. I think it's assumed that if node level verification can be performed, it should, but that most administrative policy would happen above that level.

Contributor

smarterclayton commented Mar 14, 2016

Here's the approach we were going to take in OpenShift (where we have integrated the registry into the platform), allowing the following sets of rules:

(in the rules below, "use" means "can be the input to a pod run, a build, or be tagged into the system for reuse")

  1. Only use images on the platform that have a set of known "base layers" (as mentioned above), defined by a specific policy
  2. Only use images on the platform from a whitelist/blacklist of known registries
  3. Only allow images to be used that come from the internal registry (so all use requires the import action)
  4. Only allow images to be used that have a certain label or annotation (only settable by admins) - that annotation could be anything, but typical use cases would be an admin scanning process that flags images that have been scanned. Until they're scanned, the platform will refuse to create pods that use those images
  5. Require images to have an attached / detached signature (really just a special case of 4 above) before being used
  6. Limit images that are older than a certain date from being run
  7. Perform a call to a remote system prior to allowing an image to be run

We're still trying to gather additional use cases, but most of them here are at the platform level, not at the node level. I think it's assumed that if node level verification can be performed, it should, but that most administrative policy would happen above that level.

@erictune

This comment has been minimized.

Show comment
Hide comment
@erictune

erictune Mar 23, 2016

Member

@pdbogen do you have an automated docker image build pipeline which has they keys to push to your trusted repo? If so, could that automated process be the thing responsible for ensuring that the FROM of the Dockerfile is an allowed base layer?

Member

erictune commented Mar 23, 2016

@pdbogen do you have an automated docker image build pipeline which has they keys to push to your trusted repo? If so, could that automated process be the thing responsible for ensuring that the FROM of the Dockerfile is an allowed base layer?

@erictune

This comment has been minimized.

Show comment
Hide comment
@erictune

erictune Mar 23, 2016

Member

I'm trying to separate this problem into two parts: "trusted images are built from trusted sources/bases" and "trusted containers only run trusted images".

Member

erictune commented Mar 23, 2016

I'm trying to separate this problem into two parts: "trusted images are built from trusted sources/bases" and "trusted containers only run trusted images".

@smarterclayton

This comment has been minimized.

Show comment
Hide comment
@smarterclayton

smarterclayton Mar 23, 2016

Contributor

Operational teams may delegate access to build images to users, but still
control the trusted bases. We hear that a lot. So a trusted image is only
trusted because it comes from a trusted base.

On Tue, Mar 22, 2016 at 8:15 PM, Eric Tune notifications@github.com wrote:

I'm trying to separate this problem into two parts: "trusted images are
built from trusted sources/bases" and "trusted containers only run trusted
images".


You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#22888 (comment)

Contributor

smarterclayton commented Mar 23, 2016

Operational teams may delegate access to build images to users, but still
control the trusted bases. We hear that a lot. So a trusted image is only
trusted because it comes from a trusted base.

On Tue, Mar 22, 2016 at 8:15 PM, Eric Tune notifications@github.com wrote:

I'm trying to separate this problem into two parts: "trusted images are
built from trusted sources/bases" and "trusted containers only run trusted
images".


You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#22888 (comment)

@quinton-hoole

This comment has been minimized.

Show comment
Hide comment
@quinton-hoole

quinton-hoole Apr 7, 2016

Member

@erictune Could you add an appropriate priority label?

Member

quinton-hoole commented Apr 7, 2016

@erictune Could you add an appropriate priority label?

@erictune

This comment has been minimized.

Show comment
Hide comment
@erictune

erictune Apr 13, 2016

Member

@smarterclayton

Are you planning to have an "Image Policy" object that is stored in the apiserver?

You said in item 7 above that you would Perform a call to a remote system prior to allowing an image to be run. Are you guys envisioning an admission control plugin in apiserver that calls out to an "image approver" service? Or a kubelet or docker call to an image approval system? Or some other mechanism?

For item 7, what info goes to the remote system? Image name? What else?

Do you have a use case for overriding the policy in an emergency? Would an admin role be able to bypass these checks? If so, how is username or role propagated to the point of enforcement?

Member

erictune commented Apr 13, 2016

@smarterclayton

Are you planning to have an "Image Policy" object that is stored in the apiserver?

You said in item 7 above that you would Perform a call to a remote system prior to allowing an image to be run. Are you guys envisioning an admission control plugin in apiserver that calls out to an "image approver" service? Or a kubelet or docker call to an image approval system? Or some other mechanism?

For item 7, what info goes to the remote system? Image name? What else?

Do you have a use case for overriding the policy in an emergency? Would an admin role be able to bypass these checks? If so, how is username or role propagated to the point of enforcement?

@pdbogen

This comment has been minimized.

Show comment
Hide comment
@pdbogen

pdbogen Apr 14, 2016

@erictune belatedly (sorry about that); we do a lot of automated builds with Jenkins, though the security story with Jenkins isn't great. The gist of your suggestion is that it'd definitely be easier on Kubernetes if it needed only to be a link in a trust chain- i.e., trust a registry that trusts a build pipeline, etc.; and there's definitely validity there from the k8s engineering perspective.

In our case (which I suspect is reflective of the situation of the long tail of K8S users), establishing trust throughout the entire build process is difficult, whereas Kubernetes represents a place where it can be done once, and, if done right, be done effectively.

If I were to have built a trusted pipeline, it would be largely wasted effort since Kubernetes could still be directed to run an image originating from elsewhere. Knowing that, I would heretofore have no reason to embark on such work.

Whereas if Kubernetes were the hypothetically only place I could build trust, it would be effective. If Kubernetes will only run images that pass some authorizer, then it doesn't matter where the candidate images come from; letting me secure my registry and build pipeline as an exercise in defense-in-depth.)

tl;dr: Our pipeline isn't trusted, and I appreciate the benefits of Kubernetes being just a link in a trust chain, but I think implementing the security more directly in K8S would be effective and powerful.

pdbogen commented Apr 14, 2016

@erictune belatedly (sorry about that); we do a lot of automated builds with Jenkins, though the security story with Jenkins isn't great. The gist of your suggestion is that it'd definitely be easier on Kubernetes if it needed only to be a link in a trust chain- i.e., trust a registry that trusts a build pipeline, etc.; and there's definitely validity there from the k8s engineering perspective.

In our case (which I suspect is reflective of the situation of the long tail of K8S users), establishing trust throughout the entire build process is difficult, whereas Kubernetes represents a place where it can be done once, and, if done right, be done effectively.

If I were to have built a trusted pipeline, it would be largely wasted effort since Kubernetes could still be directed to run an image originating from elsewhere. Knowing that, I would heretofore have no reason to embark on such work.

Whereas if Kubernetes were the hypothetically only place I could build trust, it would be effective. If Kubernetes will only run images that pass some authorizer, then it doesn't matter where the candidate images come from; letting me secure my registry and build pipeline as an exercise in defense-in-depth.)

tl;dr: Our pipeline isn't trusted, and I appreciate the benefits of Kubernetes being just a link in a trust chain, but I think implementing the security more directly in K8S would be effective and powerful.

@smarterclayton

This comment has been minimized.

Show comment
Hide comment
@smarterclayton

smarterclayton Apr 15, 2016

Contributor
Contributor

smarterclayton commented Apr 15, 2016

@erictune

This comment has been minimized.

Show comment
Hide comment
@erictune

erictune Apr 15, 2016

Member

Summarizing conversation so far in my own words:

  • Want a whitelist/blacklist of known registries.
  • Want rkt trust / docker content trust for any registries used. Does not provide user/namespace granularity, so can't be only solution/
  • Important that images have trusted base (FROM in Dockerfile). Transitive check needed.
  • While you could have a trusted build process that is the only thing allowed to build images, and which ensure the trusted base property, this isn't openshift and other users on this thread are thinking about the problem. For example, some users want to delegate building to devs, at least in some cases. (I guess this requires scanning images at a binary/file level?)
  • Need a way to scan and label images after build time (vulnerability, etc).
  • Need a way to refuse to run images that are not scanned.
  • Openshift plans support making a call to a remote system prior to allowing an image to be run.

Questions I still have:

  • what attributes would go to a remote system? User, namespace, image name, what else?
  • some users would like an integrated solution. Should we implement the policy as objects within kubernetes (like we plan to do with PodSecurityPolicy),
  • what happens if an image's status changes after it hits a kubelet? After RC using it is created? What are expected failure modes and surfacing?
  • need Acting-As support to propagate user name from controllers to pod creation so username can be checked when creating pod?
Member

erictune commented Apr 15, 2016

Summarizing conversation so far in my own words:

  • Want a whitelist/blacklist of known registries.
  • Want rkt trust / docker content trust for any registries used. Does not provide user/namespace granularity, so can't be only solution/
  • Important that images have trusted base (FROM in Dockerfile). Transitive check needed.
  • While you could have a trusted build process that is the only thing allowed to build images, and which ensure the trusted base property, this isn't openshift and other users on this thread are thinking about the problem. For example, some users want to delegate building to devs, at least in some cases. (I guess this requires scanning images at a binary/file level?)
  • Need a way to scan and label images after build time (vulnerability, etc).
  • Need a way to refuse to run images that are not scanned.
  • Openshift plans support making a call to a remote system prior to allowing an image to be run.

Questions I still have:

  • what attributes would go to a remote system? User, namespace, image name, what else?
  • some users would like an integrated solution. Should we implement the policy as objects within kubernetes (like we plan to do with PodSecurityPolicy),
  • what happens if an image's status changes after it hits a kubelet? After RC using it is created? What are expected failure modes and surfacing?
  • need Acting-As support to propagate user name from controllers to pod creation so username can be checked when creating pod?
@erictune

This comment has been minimized.

Show comment
Hide comment
@erictune

erictune Jun 9, 2016

Member

Concrete proposal at #27129

Member

erictune commented Jun 9, 2016

Concrete proposal at #27129

@Q-Lee

This comment has been minimized.

Show comment
Hide comment
@Q-Lee

Q-Lee Aug 10, 2016

Contributor

Proposal (#27129) is checked in and webhook API (#30241) is lgtm'ed.

@ecordell is working on the implementation of the webhook. I have some spare cycles, so let me know if there's anything I can do to nudge things along.

Contributor

Q-Lee commented Aug 10, 2016

Proposal (#27129) is checked in and webhook API (#30241) is lgtm'ed.

@ecordell is working on the implementation of the webhook. I have some spare cycles, so let me know if there's anything I can do to nudge things along.

@smarterclayton

This comment has been minimized.

Show comment
Hide comment
@smarterclayton

smarterclayton Aug 10, 2016

Contributor

And OpenShift plans on implementing the remote hook for this at some point
to point to our new image policy engine
https://github.com/smarterclayton/origin/blob/20743324d7fbe6a1090a7a3a5641f5f2e0d61527/pkg/image/admission/imagepolicy/api/v1/types.go
Probably not until the 1.5 or 1.6 timeframes though.

On Wed, Aug 10, 2016 at 4:30 PM, Q-Lee notifications@github.com wrote:

Proposal (#27129 #27129)
is checked in and webhook API (#30241
#30241) is lgtm'ed.

@ecordell https://github.com/ecordell is working on the implementation
of the webhook. I have some spare cycles, so let me know if there's
anything I can do to nudge things along.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#22888 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABG_pw6HYIxEdHWeel3l0DBF8Z8bh1Ynks5qejTUgaJpZM4HvLIL
.

Contributor

smarterclayton commented Aug 10, 2016

And OpenShift plans on implementing the remote hook for this at some point
to point to our new image policy engine
https://github.com/smarterclayton/origin/blob/20743324d7fbe6a1090a7a3a5641f5f2e0d61527/pkg/image/admission/imagepolicy/api/v1/types.go
Probably not until the 1.5 or 1.6 timeframes though.

On Wed, Aug 10, 2016 at 4:30 PM, Q-Lee notifications@github.com wrote:

Proposal (#27129 #27129)
is checked in and webhook API (#30241
#30241) is lgtm'ed.

@ecordell https://github.com/ecordell is working on the implementation
of the webhook. I have some spare cycles, so let me know if there's
anything I can do to nudge things along.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#22888 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABG_pw6HYIxEdHWeel3l0DBF8Z8bh1Ynks5qejTUgaJpZM4HvLIL
.

This was referenced Aug 19, 2016

k8s-merge-robot added a commit that referenced this issue Aug 23, 2016

Merge pull request #30923 from Q-Lee/configure-webhook
Automatic merge from submit-queue

Configure webhook

**What this PR does / why we need it**: this configures the image policy webhook + admission controller for gce/gci.

addresses: #22888

**Release note**:
```Configure image verification admission controller and webhook on gce.
```
@philips

This comment has been minimized.

Show comment
Hide comment
@philips

philips Dec 16, 2016

Contributor

@erictune @ecordell this is done and can be closed, right?

Contributor

philips commented Dec 16, 2016

@erictune @ecordell this is done and can be closed, right?

@ecordell

This comment has been minimized.

Show comment
Hide comment
@ecordell

ecordell Dec 16, 2016

Contributor

@philips Yes, this is done and went out alpha in 1.4

Contributor

ecordell commented Dec 16, 2016

@philips Yes, this is done and went out alpha in 1.4

@Q-Lee

This comment has been minimized.

Show comment
Hide comment
@Q-Lee

Q-Lee Dec 19, 2016

Contributor

Yes, this is out as an alpha feature.

Contributor

Q-Lee commented Dec 19, 2016

Yes, this is out as an alpha feature.

@krmayankk

This comment has been minimized.

Show comment
Hide comment
@krmayankk

krmayankk Mar 13, 2017

Contributor

Is there documentation on this feature someone can point us to ?

Contributor

krmayankk commented Mar 13, 2017

Is there documentation on this feature someone can point us to ?

@lumjjb

This comment has been minimized.

Show comment
Hide comment
@lumjjb

This comment has been minimized.

Show comment
Hide comment
@lumjjb

lumjjb Mar 22, 2017

We have been exploring enabling Docker Image Signing in our clusters and we looked into the Image Provenance Architecture (https://github.com/kubernetes/community/blob/master/contributors/design-proposals/image-provenance.md) in the process. We believe that there is a use-case in which we can augment the architechture for two additional features.

The two additional "features" we have identified are:

  • Image Tag Rewriting
  • ImagePullSecrets propagation

Image Tag Rewriting

This feature comes about from an issue highlighted in the architecture documents as follows;

The Backend needs to be able to resolve tags to IDs (by talking to the images repo). If the Backend resolves tags to IDs, there is some risk that the tag-to-ID mapping will be modified after approval by the Backend, but before Kubelet pulls the image. We will not address this race condition at this time.

By performing image rewriting, for example, the image ubuntu:xenial would be rewritten to use the image ID instead, ubuntu@sha256:71cd81252a3563a03ad8daee81047b62ab5d892ebbfbf71cf53415f29c130950. This allows the docker daemon to enforce the image being pulled from the notary server.

This concern stems from the scenario where an attacker has compromised a registry server. In this case, we believe it is feasible for the attacker (via fingerprinting), to determine a request from a webhook verifier from a kubelet pulling the image. In that case, the malicious attacker is able to always serve the correct image to the verifier and the tampered image to the kubelet.

To do this, the webhook would have to return a list of image names to rewrite. i.e. it would have to return a list for pod.Spec.InitContainers and a list for pod.Spec.Containers. This could potentially be added to the struct ImageReviewStatus in pkg/apis/imagepolicy/v1alpha1.

ImagePullSecrets Propagation

This feature comes about from the possibility that the image that the user wants to use is part of a private repository. Most commonly, auth details are stored in ImagePullSecrets. We think that it would be helpful for the Webhook to have access to such information.

A preliminary implementation idea would be to initialize the admission controller with an internalclientset.Interface, retrieve the ImagePullSecrets and augment the sent parameters in the struct ImageReviewSpec in pkg/apis/imagepolicy/v1alpha1.

(I'm not too sure if there is a better way to propagate this information)

lumjjb commented Mar 22, 2017

We have been exploring enabling Docker Image Signing in our clusters and we looked into the Image Provenance Architecture (https://github.com/kubernetes/community/blob/master/contributors/design-proposals/image-provenance.md) in the process. We believe that there is a use-case in which we can augment the architechture for two additional features.

The two additional "features" we have identified are:

  • Image Tag Rewriting
  • ImagePullSecrets propagation

Image Tag Rewriting

This feature comes about from an issue highlighted in the architecture documents as follows;

The Backend needs to be able to resolve tags to IDs (by talking to the images repo). If the Backend resolves tags to IDs, there is some risk that the tag-to-ID mapping will be modified after approval by the Backend, but before Kubelet pulls the image. We will not address this race condition at this time.

By performing image rewriting, for example, the image ubuntu:xenial would be rewritten to use the image ID instead, ubuntu@sha256:71cd81252a3563a03ad8daee81047b62ab5d892ebbfbf71cf53415f29c130950. This allows the docker daemon to enforce the image being pulled from the notary server.

This concern stems from the scenario where an attacker has compromised a registry server. In this case, we believe it is feasible for the attacker (via fingerprinting), to determine a request from a webhook verifier from a kubelet pulling the image. In that case, the malicious attacker is able to always serve the correct image to the verifier and the tampered image to the kubelet.

To do this, the webhook would have to return a list of image names to rewrite. i.e. it would have to return a list for pod.Spec.InitContainers and a list for pod.Spec.Containers. This could potentially be added to the struct ImageReviewStatus in pkg/apis/imagepolicy/v1alpha1.

ImagePullSecrets Propagation

This feature comes about from the possibility that the image that the user wants to use is part of a private repository. Most commonly, auth details are stored in ImagePullSecrets. We think that it would be helpful for the Webhook to have access to such information.

A preliminary implementation idea would be to initialize the admission controller with an internalclientset.Interface, retrieve the ImagePullSecrets and augment the sent parameters in the struct ImageReviewSpec in pkg/apis/imagepolicy/v1alpha1.

(I'm not too sure if there is a better way to propagate this information)

@ecordell

This comment has been minimized.

Show comment
Hide comment
@ecordell

ecordell Mar 22, 2017

Contributor

W.R.T. Image Tag Rewriting, I know this is being discussed but I'm not sure where it's written down. A decision needs to be made about where the translation point happens, and for consistency reasons that probably needs to be before the image name reaches the kubelet for running.

Please see #31524 for a discussion of ImagePullSecret propagation. There are a lot of concerns with it, and I've convinced myself that something like an oauth flow (that kubernetes doesn't need to know about) can solve the problem. I'd love to hear different ideas if you have them!

Contributor

ecordell commented Mar 22, 2017

W.R.T. Image Tag Rewriting, I know this is being discussed but I'm not sure where it's written down. A decision needs to be made about where the translation point happens, and for consistency reasons that probably needs to be before the image name reaches the kubelet for running.

Please see #31524 for a discussion of ImagePullSecret propagation. There are a lot of concerns with it, and I've convinced myself that something like an oauth flow (that kubernetes doesn't need to know about) can solve the problem. I'd love to hear different ideas if you have them!

@lumjjb

This comment has been minimized.

Show comment
Hide comment
@lumjjb

lumjjb Mar 23, 2017

Thanks @ecordell for the feedback! By the way, do you have a image policy webhook that I can take a look at It would be great if there is something that I can play around with!

W.R.T. Image Tag Rewriting, I know this is being discussed but I'm not sure where it's written down. A decision needs to be made about where the translation point happens, and for consistency reasons that probably needs to be before the image name reaches the kubelet for running.

Agreed! It would be nice if somehow I can view/jump in on this discussion. I've played around with admission controller where I did rewriting at the admission controller level, which seems like an OK idea (imo).

for i := range pod.Spec.Containers {
    if pod.Spec.Containers[i].Image != "kube-aggregator" {
        newImg, err :=  rewriteImageString(pod.Spec.Containers[i].Image)
        if err != nil {
            return admission.NewForbidden(attributes, errors.New("IMG_SIGN failed to verify notary"))
        }
        pod.Spec.Containers[i].Image = newImg
    }
}

Please see #31524 for a discussion of ImagePullSecret propagation. There are a lot of concerns with it, and I've convinced myself that something like an oauth flow (that kubernetes doesn't need to know about) can solve the problem. I'd love to hear different ideas if you have them!

I have do delve a bit deeper into the discussion, but it seems like there's a differentiation between use cases of ImagePullSecrets VS Non-ImagePullSecrets and External Party VS Extension of the Cluster.

EDIT: Reading through the ImagePullSecrets thread: I guessed I only looked at the "ImagePullSecrets" case.

I think oauth is a nice framework to do things. I will have to look into closer details. The registries and notary servers already support the oauth framework to perform the delegation, which is nice. :).

Just for sharing: In playing around with this, I had a very simplistic view on this, in terms of just extracting the secrets and passing it along, in the same way that the kubelet would consume the ImagePullSecret (We assume the admission controller/webhook is just an extension of k8). And just passing that along in the ImageReviewSpec.

aCore := a.client.Core()
secrets:= aCore.Secrets(pod.Namespace)
if len(pod.Spec.ImagePullSecrets) > 0 {
    sec, err := secrets.Get(pod.Spec.ImagePullSecrets[0].Name, metav1.GetOptions{})
}

lumjjb commented Mar 23, 2017

Thanks @ecordell for the feedback! By the way, do you have a image policy webhook that I can take a look at It would be great if there is something that I can play around with!

W.R.T. Image Tag Rewriting, I know this is being discussed but I'm not sure where it's written down. A decision needs to be made about where the translation point happens, and for consistency reasons that probably needs to be before the image name reaches the kubelet for running.

Agreed! It would be nice if somehow I can view/jump in on this discussion. I've played around with admission controller where I did rewriting at the admission controller level, which seems like an OK idea (imo).

for i := range pod.Spec.Containers {
    if pod.Spec.Containers[i].Image != "kube-aggregator" {
        newImg, err :=  rewriteImageString(pod.Spec.Containers[i].Image)
        if err != nil {
            return admission.NewForbidden(attributes, errors.New("IMG_SIGN failed to verify notary"))
        }
        pod.Spec.Containers[i].Image = newImg
    }
}

Please see #31524 for a discussion of ImagePullSecret propagation. There are a lot of concerns with it, and I've convinced myself that something like an oauth flow (that kubernetes doesn't need to know about) can solve the problem. I'd love to hear different ideas if you have them!

I have do delve a bit deeper into the discussion, but it seems like there's a differentiation between use cases of ImagePullSecrets VS Non-ImagePullSecrets and External Party VS Extension of the Cluster.

EDIT: Reading through the ImagePullSecrets thread: I guessed I only looked at the "ImagePullSecrets" case.

I think oauth is a nice framework to do things. I will have to look into closer details. The registries and notary servers already support the oauth framework to perform the delegation, which is nice. :).

Just for sharing: In playing around with this, I had a very simplistic view on this, in terms of just extracting the secrets and passing it along, in the same way that the kubelet would consume the ImagePullSecret (We assume the admission controller/webhook is just an extension of k8). And just passing that along in the ImageReviewSpec.

aCore := a.client.Core()
secrets:= aCore.Secrets(pod.Namespace)
if len(pod.Spec.ImagePullSecrets) > 0 {
    sec, err := secrets.Get(pod.Spec.ImagePullSecrets[0].Name, metav1.GetOptions{})
}
@lumjjb

This comment has been minimized.

Show comment
Hide comment
@lumjjb

lumjjb Jun 5, 2017

@krmayankk, @ecordell

I have implemented the image rewriting on the admission controller: https://github.com/kubernetes/kubernetes/compare/master...lumjjb:img-webhook-con-rewrite?expand=1

Do you think we should have this be merged? Or should we discuss this more?

This addresses the problem in the documentation (https://github.com/kubernetes/community/blob/master/contributors/design-proposals/image-provenance.md ):

The Backend needs to be able to resolve tags to IDs (by talking to the images repo). If the Backend resolves tags to IDs, there is some risk that the tag-to-ID mapping will be modified after approval by the Backend, but before Kubelet pulls the image. We will not address this race condition at this time.

However this is not just a race, since it is possible for a malicious actor to probably fingerprint whether it is the docker agent or webhook that is querying for an image (and provide the tampered response only to the docker agent running on the k8s worker).

lumjjb commented Jun 5, 2017

@krmayankk, @ecordell

I have implemented the image rewriting on the admission controller: https://github.com/kubernetes/kubernetes/compare/master...lumjjb:img-webhook-con-rewrite?expand=1

Do you think we should have this be merged? Or should we discuss this more?

This addresses the problem in the documentation (https://github.com/kubernetes/community/blob/master/contributors/design-proposals/image-provenance.md ):

The Backend needs to be able to resolve tags to IDs (by talking to the images repo). If the Backend resolves tags to IDs, there is some risk that the tag-to-ID mapping will be modified after approval by the Backend, but before Kubelet pulls the image. We will not address this race condition at this time.

However this is not just a race, since it is possible for a malicious actor to probably fingerprint whether it is the docker agent or webhook that is querying for an image (and provide the tampered response only to the docker agent running on the k8s worker).

@ecordell

This comment has been minimized.

Show comment
Hide comment
@ecordell

ecordell Jun 6, 2017

Contributor

@lumjjb

I sketched this out a couple of weeks ago with a slightly different approach:
master...ecordell:imagereviewwebhook-digest

(I didn't flesh out the tests or do the codegen)

I was concerned that anything we do here would be usurped by kubernetes/community#132, which intends to rework admission controllers and explicitly avoid mutating webhook backends for the first version.

However this is not just a race, since it is possible for a malicious actor to probably fingerprint whether it is the docker agent or webhook that is querying for an image (and provide the tampered response only to the docker agent running on the k8s worker).

This is true, but I think you're just highlighting that any admission controller that can mutate the request (including a webhook backend in this case) needs to be as trusted and as secured as any of the rest of the control plane.

Contributor

ecordell commented Jun 6, 2017

@lumjjb

I sketched this out a couple of weeks ago with a slightly different approach:
master...ecordell:imagereviewwebhook-digest

(I didn't flesh out the tests or do the codegen)

I was concerned that anything we do here would be usurped by kubernetes/community#132, which intends to rework admission controllers and explicitly avoid mutating webhook backends for the first version.

However this is not just a race, since it is possible for a malicious actor to probably fingerprint whether it is the docker agent or webhook that is querying for an image (and provide the tampered response only to the docker agent running on the k8s worker).

This is true, but I think you're just highlighting that any admission controller that can mutate the request (including a webhook backend in this case) needs to be as trusted and as secured as any of the rest of the control plane.

@smarterclayton

This comment has been minimized.

Show comment
Hide comment
@smarterclayton

smarterclayton Jun 6, 2017

Contributor
Contributor

smarterclayton commented Jun 6, 2017

@lumjjb

This comment has been minimized.

Show comment
Hide comment
@lumjjb

lumjjb Jun 6, 2017

@ecordell

I sketched this out a couple of weeks ago with a slightly different approach:
master...ecordell:imagereviewwebhook-digest

I like your dictionary implementation more actually. Looks a lot neater.

I wrote mine in consideration that there may be more things in the container spec in the future that we may require rewriting, but seems like there isn't a usecase in sight at the moment.

I was concerned that anything we do here would be usurped by kubernetes/community#132, which intends to rework admission controllers and explicitly avoid mutating webhook backends for the first version.

Given the new design, would the image policy admission controller be an Initializer or a Mutating Admission Webhook? (I am guessing the later, but there's not too much details on this, so it seems unclear).

Also, should we update the document to express the intention of the imagepolicy/admission.go to also perform mutation since this is something that we want?

For the current build i suppose it could be possible to add a flag in the config file to "trust" the webhook to perform rewriting or disallow the rewriting (as you pointed out, this still won't work with the new design immediately if it is an Admission Webhook), so perhaps it may be a moot point to introduce the feature now and take it out again until 1.8 unless there will be support of the "old" admission controllers.

@smarterclayton It would be nice to be able to get this in a little sooner than that if possible :). Understand that there may be a need to rewrite the code/redesign parts of it later.

lumjjb commented Jun 6, 2017

@ecordell

I sketched this out a couple of weeks ago with a slightly different approach:
master...ecordell:imagereviewwebhook-digest

I like your dictionary implementation more actually. Looks a lot neater.

I wrote mine in consideration that there may be more things in the container spec in the future that we may require rewriting, but seems like there isn't a usecase in sight at the moment.

I was concerned that anything we do here would be usurped by kubernetes/community#132, which intends to rework admission controllers and explicitly avoid mutating webhook backends for the first version.

Given the new design, would the image policy admission controller be an Initializer or a Mutating Admission Webhook? (I am guessing the later, but there's not too much details on this, so it seems unclear).

Also, should we update the document to express the intention of the imagepolicy/admission.go to also perform mutation since this is something that we want?

For the current build i suppose it could be possible to add a flag in the config file to "trust" the webhook to perform rewriting or disallow the rewriting (as you pointed out, this still won't work with the new design immediately if it is an Admission Webhook), so perhaps it may be a moot point to introduce the feature now and take it out again until 1.8 unless there will be support of the "old" admission controllers.

@smarterclayton It would be nice to be able to get this in a little sooner than that if possible :). Understand that there may be a need to rewrite the code/redesign parts of it later.

@lumjjb

This comment has been minimized.

Show comment
Hide comment
@lumjjb

lumjjb Jul 10, 2017

@smarterclayton Is there anything we can do to help on the work on mutating webhook admission webhooks? Would definitely like to see it in 1.8!

lumjjb commented Jul 10, 2017

@smarterclayton Is there anything we can do to help on the work on mutating webhook admission webhooks? Would definitely like to see it in 1.8!

@fejta-bot

This comment has been minimized.

Show comment
Hide comment
@fejta-bot

fejta-bot Jan 6, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

fejta-bot commented Jan 6, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

@fejta-bot

This comment has been minimized.

Show comment
Hide comment
@fejta-bot

fejta-bot Feb 9, 2018

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

fejta-bot commented Feb 9, 2018

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

@fejta-bot

This comment has been minimized.

Show comment
Hide comment
@fejta-bot

fejta-bot Mar 12, 2018

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

fejta-bot commented Mar 12, 2018

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment