Kubelet should enforce access to images after they've already been pulled to a node

[ncdc]

Problem statement

Some images require pull secrets in order to be used by the Kubelet. Pull secrets are per namespace. Imagine the following scenario:

1. If a pod in namespaceA has valid pull secrets for foo/bar:stable, when it is scheduled to a given node, the Kubelet uses the pull secrets to pull the image to the node
2. If a pod in namespaceB also refers to foo/bar:stable but does not have valid pull secrets for it, if it happens to be scheduled to the same node as the pod in step 1, the Kubelet respects the image pull policy
3. Because the image already exists on the node, if the pull policy is Never or IfNotPresent, the Kubelet runs the pod, even though namespaceB shouldn't be able to use that image.

Proposed solutions

Option 1 - Kubelet flag to ignore pull policy and always try to pull

Add a flag to the kubelet, --always-pull-images (or something similarly named) that, if true, would force the Kubelet to attempt to pull every image prior to running a pod, regardless of the image's pull policy.

Pros:

• Easy to implement
• Guarantees that a namespace is allowed to use an image

Cons:

• Overrides user-specified intent in image pull policy
• Requires remote registry to be online to check authorization
• Introduces small amount of latency to pod startup time (because of the additional network requests to the registry)

Option 2 - Keep an in-memory cache of which namespaces are allowed to use which images

After an image has been successfully pulled, record the mapping from namespace to image in an in-memory cache in the Kubelet. Subsequent attempts by pods in the same namespace to use the same image would respect the image pull policy. Any attempts by pods in other namespaces to use this image would ignore the image pull policy and instead force a pull attempt.

Pros:

• Doesn't override user-specified intent in image pull policy
• No added latency to pod startup time

Cons:

• Doesn't handle authorization revocation; if the credentials to pull an image have changed in the remote registry, there is no way to evict an entry from the cache, possibly allowing access to an image that would otherwise fail with option 1. This could be mitigated if the entries in the cache have a TTL.

Suggested path forward

Looking for feedback from the community on this one. I'd like to implement option 1 in the short term, with the default being false (so image pull policy is respected) to keep the status quo. If we can come to an agreement on option 2, I'd like to move to that eventually.

@kubernetes/goog-node @kubernetes/rh-cluster-infra @kubernetes/sig-node

[dchen1107]

@ncdc I agreed we should enforce the image access / usage after the image is pulled, and haven't put much thoughts on how to achieve that yet.

But I don't think option 1 is the right approach here because all cons you listed there, especially the first one: Overrides user-specified intent in image pull policy. Assuming the user specify the pullPolicy as Never or IfNotPresent, they expect when the container is restarted with the same version / tag of the image, the behavior should be the same. The option 1 actually breaks that, and bring a lot of unpredictability to the overall system.

[ncdc]

@dchen1107 if you really want to ensure your running images don't change, you should be referring to them by their content-addressable ID. In OpenShift, we have ImageStream resources which store metadata about tags and images, and our resources for performing builds and deployments use the information in the image streams to translate a pull spec such as foo/bar:latest into foo/bar@sha256:... so that the system knows the exact image that must be used, and you don't face the possibility of a moving tag.

Another option here would be to issue the same HTTP requests to retrieve the image manifest from the registry that would be issued during a real pull, to ensure that the credentials are still valid. If the request succeeds, the container creation can succeed with whatever image already exists on the node for the pull spec in question.

[ncdc]

cc @smarterclayton

[smarterclayton]

Our concern is that this is seen as a security risk in regular production use of Kubernetes for certain organizations that have what they consider secret images. We're looking for the most orthogonal solution that gets them over the hump so they aren't blocked by this. There definitely needs to be a real solution, but I think pull always is similar in many ways to our other kubelet policies.

[ncdc]

@dchen1107 would you at least accept a PR that adds a flag to enable pull always, defaulted to false?

[vishh]

Is updating user configs to be PullAlways an option here?
Reaching out the registry directly will help with other use cases as well like image size feasibility checks. So I'm in favor of that approach.

[ncdc]

@vishh updating in what way? As in hoping the users will do it themselves? Or via an admission controller?

[vishh]

Ideally, users update their configs. It that is not an option, may be via a
custom admission controller?

On Fri, Dec 18, 2015 at 11:51 AM, Andy Goldstein notifications@github.com
wrote:

@vishh https://github.com/vishh updating in what way? As in hoping the
users will do it themselves? Or via an admission controller?

—
Reply to this email directly or view it on GitHub
#18787 (comment)
.

[ncdc]

Yeah, we can't rely on users, sadly. If a flag is unpalatable, would you all take this as an admission controller (off by default)?

[vishh]

Personally, admission controller seems to be the least hacky solution. So
SGTM. cc @davidopp

On Fri, Dec 18, 2015 at 11:56 AM, Andy Goldstein notifications@github.com
wrote:

Yeah, we can't rely on users, sadly. If a flag is unpalatable, would you
all take this as an admission controller (off by default)?

—
Reply to this email directly or view it on GitHub
#18787 (comment)
.

[smarterclayton]

Is having Kubelet policy considered hacky? What's the difference between this and saying whether the kubelet accepts privileged pods?

[vishh]

@smarterclayton: By changing the config, we are not overriding the API. That was the rationale.