Force pods to re-pull an image without changing the image tag

[yissachar]

Problem

A frequent question that comes up on Slack and Stack Overflow is how to trigger an update to a Deployment/RS/RC when the image tag hasn't changed but the underlying image has.

Consider:

1. There is an existing Deployment with image foo:latest
2. User builds a new image foo:latest
3. User pushes foo:latest to their registry
4. User wants to do something here to tell the Deployment to pull the new image and do a rolling-update of existing pods

The problem is that there is no existing Kubernetes mechanism which properly covers this.

Current Workarounds

• Always change the image tag when deploying a new version
• Refer to the image hash instead of tag, e.g. localhost:5000/andy/busybox@sha256:2aac5e7514fbc77125bd315abe9e7b0257db05fe498af01a58e239ebaccf82a8
• Use latest tag or imagePullPolicy: Always and delete the pods. New pods will pull the new image. This approach doesn't do a rolling update and will result in downtime.
• Fake a change to the Deployment by changing something other than the image

Possible Solutions

• Rolling restart of pods #13488 If rolling restart were implemented, users could do a rolling-restart to pull the new image.
• Have a controller that watches the image registry and automatically updates the Deployment to use the latest image hash for a given tag. See Image name/tag resolution preprocessing pass #1697 (comment)

cc @justinsb

[justinsb]

This is indeed important, and I think there are two cases:

1. when we have a full CI system like Jenkins (aka “do I really have to use sed”)
2. we have a limited system like dockerhub that only re-tags latest

[yujuhong]

@yissachar using :latest tag IMO is not the best practice as it's hard to track what image is really in use in your pod. I think tagging images by versions or using the digests is strictly better than reusing the same tag. Is it really such a hassle to do that?

[Arachnid]

@yujuhong Sometimes it's very useful to be able to do this. For instance, we run a testing cluster that should run a build from the latest commit on the master branch of our repository. There aren't tags or branches for every commit, so ':latest' is the logical and most practical name for it.

Wouldn't it make more sense if Kubernetes stored and checked the hash of the deployed container instead of its (mutable) name anyway, though?

[yissachar]

@yujuhong I agree that if you can do so then you should (and I do!). But this question comes up quite frequently and often users cannot easily tag every build (this often arises with CI systems). They need a solution with less friction to their process, and this means they want to see some way of updating a Deployment without changing the image tag.

[dominiek]

I am running into the same limitations. I agree that in an ideal setup every version would be explicitly tagged, but this can be cumbersome in highly automated environments. Think of dozens of containers with 100 new versions per day.

Also, when debugging and setting up a new infrastructure there are a lot of small tweaks made to the containers. Having a force-repull on a Deployment will make the process more frictionless.

[yujuhong]

I am running into the same limitations. I agree that in an ideal setup every version would be explicitly tagged, but this can be cumbersome in highly automated environments. Think of dozens of containers with 100 new versions per day.

Hmm....I still think automatically tagging images by commit hash would be ideal, but I see that it may be difficult to do for some CI systems.

In order to do this, we'd need (1) a component to detect the change and (2) a mechanism to restart the pod.

Also, when debugging and setting up a new infrastructure there are a lot of small tweaks made to the containers. Having a force-repull on a Deployment will make the process more frictionless.

This sounds reasonable.

/cc @pwittrock, who has more context on the CI systems.

[Arachnid]

Hmm....I still think automatically tagging images by commit hash would be ideal, but I see that it may be difficult to do for some CI systems.

Creating a tag for every single commit is also pretty pointless - commits already have unique identifiers - especially when you only care about the last one.

What I don't understand is why Kubernetes treats tags as if they're immutable, when they're explicitly mutable human-readable names for immutable identifiers (the hash of the manifest).