New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Force pods to re-pull an image without changing the image tag #33664

Closed
yissachar opened this Issue Sep 28, 2016 · 78 comments

Comments

Projects
None yet
@yissachar
Copy link

yissachar commented Sep 28, 2016

Problem

A frequent question that comes up on Slack and Stack Overflow is how to trigger an update to a Deployment/RS/RC when the image tag hasn't changed but the underlying image has.

Consider:

  1. There is an existing Deployment with image foo:latest
  2. User builds a new image foo:latest
  3. User pushes foo:latest to their registry
  4. User wants to do something here to tell the Deployment to pull the new image and do a rolling-update of existing pods

The problem is that there is no existing Kubernetes mechanism which properly covers this.

Current Workarounds

  • Always change the image tag when deploying a new version
  • Refer to the image hash instead of tag, e.g. localhost:5000/andy/busybox@sha256:2aac5e7514fbc77125bd315abe9e7b0257db05fe498af01a58e239ebaccf82a8
  • Use latest tag or imagePullPolicy: Always and delete the pods. New pods will pull the new image. This approach doesn't do a rolling update and will result in downtime.
  • Fake a change to the Deployment by changing something other than the image

Possible Solutions

  • #13488 If rolling restart were implemented, users could do a rolling-restart to pull the new image.
  • Have a controller that watches the image registry and automatically updates the Deployment to use the latest image hash for a given tag. See #1697 (comment)

cc @justinsb

@justinsb

This comment has been minimized.

Copy link
Member

justinsb commented Sep 28, 2016

This is indeed important, and I think there are two cases:

  1. when we have a full CI system like Jenkins (aka “do I really have to use sed”)
  2. we have a limited system like dockerhub that only re-tags latest
@yujuhong

This comment has been minimized.

Copy link
Contributor

yujuhong commented Oct 14, 2016

@yissachar using :latest tag IMO is not the best practice as it's hard to track what image is really in use in your pod. I think tagging images by versions or using the digests is strictly better than reusing the same tag. Is it really such a hassle to do that?

@Arachnid

This comment has been minimized.

Copy link

Arachnid commented Oct 14, 2016

@yujuhong Sometimes it's very useful to be able to do this. For instance, we run a testing cluster that should run a build from the latest commit on the master branch of our repository. There aren't tags or branches for every commit, so ':latest' is the logical and most practical name for it.

Wouldn't it make more sense if Kubernetes stored and checked the hash of the deployed container instead of its (mutable) name anyway, though?

@yissachar

This comment has been minimized.

Copy link

yissachar commented Oct 14, 2016

@yujuhong I agree that if you can do so then you should (and I do!). But this question comes up quite frequently and often users cannot easily tag every build (this often arises with CI systems). They need a solution with less friction to their process, and this means they want to see some way of updating a Deployment without changing the image tag.

@dominiek

This comment has been minimized.

Copy link

dominiek commented Oct 14, 2016

I am running into the same limitations. I agree that in an ideal setup every version would be explicitly tagged, but this can be cumbersome in highly automated environments. Think of dozens of containers with 100 new versions per day.

Also, when debugging and setting up a new infrastructure there are a lot of small tweaks made to the containers. Having a force-repull on a Deployment will make the process more frictionless.

@yujuhong

This comment has been minimized.

Copy link
Contributor

yujuhong commented Oct 14, 2016

I am running into the same limitations. I agree that in an ideal setup every version would be explicitly tagged, but this can be cumbersome in highly automated environments. Think of dozens of containers with 100 new versions per day.

Hmm....I still think automatically tagging images by commit hash would be ideal, but I see that it may be difficult to do for some CI systems.

In order to do this, we'd need (1) a component to detect the change and (2) a mechanism to restart the pod.

Also, when debugging and setting up a new infrastructure there are a lot of small tweaks made to the containers. Having a force-repull on a Deployment will make the process more frictionless.

This sounds reasonable.

/cc @pwittrock, who has more context on the CI systems.

@Arachnid

This comment has been minimized.

Copy link

Arachnid commented Oct 14, 2016

Hmm....I still think automatically tagging images by commit hash would be ideal, but I see that it may be difficult to do for some CI systems.

Creating a tag for every single commit is also pretty pointless - commits already have unique identifiers - especially when you only care about the last one.

What I don't understand is why Kubernetes treats tags as if they're immutable, when they're explicitly mutable human-readable names for immutable identifiers (the hash of the manifest).

@pwittrock

This comment has been minimized.

Copy link
Member

pwittrock commented Oct 15, 2016

@erictune @janetkuo

This could live either outside the deployment in a CICD system that forces forces a new deployment rollout. Alternatively, it could be a field on the deployment. WDYT?

@alphashuro

This comment has been minimized.

Copy link

alphashuro commented Oct 18, 2016

What is the consensus on this?

@pwittrock

This comment has been minimized.

Copy link
Member

pwittrock commented Oct 18, 2016

Longer term some CICD system should support this.

Immediate term: It would probably be simple to create a controller that listens for changes to a container registry and then updates a label on all deployments with a specific annotation. You could install this controller into your kubernetes cluster using helm.

I will try to hack a prototype together later this week.

@pwittrock

This comment has been minimized.

Copy link
Member

pwittrock commented Oct 26, 2016

Quick question - why not set an annotation on the pod template with the current time to force the repull. I believe this would execute an update using the deployment's strategy to rollout the new im age.

I put together an example of how to write a controller to do this in response to webhook callbacks from dockerhub. I need to add some documentation and then will post the example here. Ideally I would put together a helm chart for this as well.

@pwittrock

This comment has been minimized.

Copy link
Member

pwittrock commented Oct 26, 2016

FYI here is a simple controller for pushing deployment updates in response to webhook callbacks. It natively supports dockerhub, but you can manually post to it from the command line.

container-image-pusher

@brendandburns

This comment has been minimized.

Copy link
Contributor

brendandburns commented Nov 4, 2016

fwiw, I don't think we should support this as a proper Kubernetes API.

I don't quite understand the use case, as I see it there are two scenarios:

  1. you are using 'latest' for testing (this is the "no sed" use case), in this case, downtime is fine, and indeed the right approach is likely to completely blow away your stack and redeploy from scratch to get a clean run.

  2. you are running 'latest' in prod. If you do this, you are just asking for trouble, there are a myriad different failure modes that can occur (and perhaps more importantly) if you aren't tagging your images with the git hash you don't have a rollback path (since the old latest has been blown away by the newer latest)

Finally, I believe that kubectl set <...> eliminates the need for sed

@justinclayton or @yissachar is there a use case I'm missing here?

@Arachnid

This comment has been minimized.

Copy link

Arachnid commented Nov 4, 2016

  1. you are using 'latest' for testing (this is the "no sed" use case), in this case, downtime is fine, and indeed the right approach is likely to completely blow away your stack and redeploy from scratch to get a clean run.

I'm not sure I follow the argument here. Downtime isn't fine in our use-case, of running monitoring nodes of the latest instances of our software. It seems sensible to be able to apply the same deployment mechanics to this as to anything else.

More broadly, docker tags, like most name services, are fundamentally mutable by design - and docker repos provide a way to resolve a tag to the current image hash. I don't understand why Kubernetes associates the mutable tag with a deployed pod, then treats it as immutable, instead of just using the immutable identifier in the first place.

Finally, I believe that kubectl set <...> eliminates the need for sed

Perhaps, but it still leaves the task of resolving tag name to image hash up to the user, something that's definitely nontrivial to do with existing tools.

@serverhorror

This comment has been minimized.

Copy link

serverhorror commented Nov 4, 2016

@brendandburns I'm interested in this as well. Not for the reasons of updating the pods.

My situation is this: Pods and Containers are pretty stable but the data moves way faster. Our data sets span 100s of GBs per file with 100s of files (genomic data, life sciences). And since a lot of the software is academic there isn't much engineering effort going into it. Currently the easiest way to "redeploy" is to replace a config map that points to the new data sets. Kubernetes takes care of replacing the actual config file in the container but right now there's no way to trigger a a kind of rolling-update so that pods get killed and restarted the same way it would happen with an update to the actual container versions. I don't want to get into the business of image management too much so I try not to update images every time data changes.

Does that makes sense?

I'm happy to go any other path, but my current experience is that this seems to be the way to go when there's not enough development bandwidth to fix the underlying issues.

@kargakis

This comment has been minimized.

Copy link
Member

kargakis commented Nov 4, 2016

#13488 seems related

@brendandburns

This comment has been minimized.

Copy link
Contributor

brendandburns commented Nov 6, 2016

@serverhorror I think the way that I would accomplish what you want is that I would set up a side car container that is in the same pod as your main container. The job of that sidecar is to monitor the config file and send a signal (e.g. SIGHUP or SIGKILL) to your main container that indicates that the data file has changed.

You could also use container health checks e.g. set up a health check for your 'main' container to point to a web service hosted by your sidecar. Whenever the sidecar changes, the health check goes 'unhealthy' and the kubelet will automatically restart your main container.

@Arachnid I guess I fundamentally believe that tags should not be used in a mutable matter. If you use image tags in a mutable way, then the definition stops having meaning, you no longer can know for sure what is running in a particular container just by looking at the API object. Docker may allow you to mutate tags on images, but I think that the Kubernetes philosophy (and hard-won experience of running containerized systems at scale) is that mutable tags (and 'latest' in particular) are very dangerous to use in a production environment.

I agree that the right thing to do is to apply the same deployment mechanics in test and in prod, given that, and the belief that latest is dangerous in production, it means the right answer is to use git-sha for your tag and use the Deployment object to do rollouts for both test and prod.

Here are some examples of concrete production issues that I ran into due to the use of latest:

  • A task restart in the middle of the night accidentally 'upgraded' one of my apps to a debug (e.g. slow) build b/c someone mistakenly moved the 'latest' tag.
  • I believed that a server was actually fully upgraded because it was running 'latest' but actually the image pull on the machine was perma-failing and so it was lagging behind on an old version.
  • Auto-scaling caused an accidental 'upgrade' because when it scaled up via 'latest' it created containers using the new image, when I wasn't ready to roll it out yet.

I hope that helps explain why I think that mutable labels are a dangerous idea.

@Arachnid

This comment has been minimized.

Copy link

Arachnid commented Nov 6, 2016

I guess I fundamentally believe that tags should not be used in a mutable matter. If you use image tags in a mutable way, then the definition stops having meaning, you no longer can know for sure what is running in a particular container just by looking at the API object.

Agreed, as-is they're dangerous, but this could be trivially resolved by having the API object retain the hash of the container image as the permanent identifier for it, rather than assuming the (mutable) tag won't change. This seems like a fundamental mismatch between how Docker treats tags and how Kubernetes treats them, but it seems resolvable, to me. Every one of the problems you list below could be resolved by storing and explicitly displaying the hash of the currently running container.

Tagging images by their git hashes doesn't really express what I mean when I create a deployment, and introduces awkward dependencies requiring me to propagate those tags through the system.

@serverhorror

This comment has been minimized.

Copy link

serverhorror commented Nov 8, 2016

@brendandburns Right, liveness checks seem to be another easy way. That is serving my needs, could have thought of that. Consider my argument for this taken back :)

@deitch

This comment has been minimized.

Copy link

deitch commented Nov 29, 2016

@brendandburns and @yujuhong: I could see this being useful in a number of use cases, where "latest" is used in prod.

"latest" is dangerous in production

Depends on how "latest" gets used. I have worked with a number of environments where there is a single image registry that supports prod/testing/etc. (which makes sense). However, the given repos can be populated only by CI. Builds off of any branch get tagged correctly with versions, but builds off HEAD from master (which pass all tests of course) also get tagged "latest".

Prod environments, in turn, point at "latest". That way I don't need to update anything about versions for prod; l just need to say, "go rolling update" (either automatically or when a human approves, which hopefully will be removed from the process very soon).

To answer the "danger" question:

  1. No human can ever mistakenly tag something latest because humans do not get to push to the repos in question.
  2. I always know what version I am running by looking at image tags.
  3. I am much more worried about letting a human update my deployment config and entering tag "1.2.3" instead of "1.2.4" (or worse "145abcd6" instead of "145adcd6") and deploying. Which is why we went "latest" in the first place.

So:

  1. I think "rolling update of a deployment without changing tags" is a real use case that real people have, worth supporting.
  2. I am more than happy to switch to immutable tags (and avoid the problem), if I can find a way to not involve humans in the tag-change (step 3) process above.

I guess I could whip up a script or Web app that lists all available tags that come from "master" and makes them pick one, and when we go full automated, have the CI also pull the deployment, update the image, and redeploy?

@alirezaDavid

This comment has been minimized.

Copy link

alirezaDavid commented Jan 9, 2017

if kubectl apply -f for-god-sake-update-latest-image.yaml can update latest image we should really happy. (with ImagePullPollicy: Always)

@dszmaj

This comment has been minimized.

Copy link

dszmaj commented May 10, 2018

@fgreg make a quick helm chart and use { .Release.Time } as a value of env in a pod and helm upgrade --install, this forces redeploy of pods and you can do it in one command without downtime and without ssh'ing into nodes or doing anything more in kube dashboard or smth

but boy it feels ugly as hell....

@OskarStark

This comment has been minimized.

Copy link

OskarStark commented May 15, 2018

thank you @spiddy this did the trick for me 👍 🎉

@mufabi

This comment has been minimized.

Copy link

mufabi commented May 30, 2018

Thanks to @dszmaj + @spiddy
I added date: "{{ .Release.Time.Seconds }}" to spec/template/metadata/labels in all of my charts.
Everytime I run helm upgrade it pulls the new latest tagged image.

@bgrant0607

This comment has been minimized.

Copy link
Member

bgrant0607 commented Jun 5, 2018

Pull by digest is supported. The recommended solution is still to use the digest rather than the tag. kubectl set image can update the field, either in the live state, or even on disk using kubectl set image --local -o yaml.

@dat-boris dat-boris referenced this issue Jun 16, 2018

Merged

Fix up makefile for build and updates #1526

2 of 2 tasks complete
@adampl

This comment has been minimized.

Copy link

adampl commented Jun 22, 2018

Thanks @spiddy, though I prefer to avoid the \, so I cooked up this :)

printf '{"spec":{"template":{"metadata":{"labels":{"date":"%s"}}}}}' `date +%s` \
       | xargs -0 kubectl patch deployment myapp -p
@AndresPineros

This comment has been minimized.

Copy link

AndresPineros commented Jul 2, 2018

For our CI-CD we have 3 use cases for developers and QA's:

  1. Change the image tag. For example, 'develop' to 'feature_testing_something'
  2. Re-pulling the same image. If they updated the 'develop' tag and want to re-deploy it.
  3. Just restarting a service. No image was updated, nothing changed. They just want to restart the pods.

So, I haven't implemented this but to automate I would do something LIKE this in my pipelines:

import subprocess
import time
import argparse

ap = argparse.ArgumentParser()
ap.add_argument('-d', '--deploy', required=True)
ap.add_argument('-c', '--container', required=True)
ap.add_argument('-i', '--image', required=True)

args = vars(ap.parse_args())

current_image=subprocess.check_output("kubectl get deployment {} -o=jsonpath='{{$.spec.template.spec.containers[:1].image}}"
            .format(args['deploy']).split(" "))[1:]

if current_image != args['image']:
    print("Case: image or tag changed")
    subprocess.call("kubectl set image deployment/{} {}={}".format(args['deploy'], args['container'], args['image']).split(" "))
else:
    print("Case: re-pulling container")
    subprocess.call("kubectl set env deployment/{} {}={}".format(args['deploy'], "K8S_FORCE", time.time()).split(" "))

And call it like:

python update_deploy.py -d my_deploy -c my_container -i new_image:some_tag

I'd use the python kubernetes client instead of subprocess though.
Yes, very ugly.

@nsidhaye

This comment has been minimized.

Copy link

nsidhaye commented Jul 24, 2018

Should we reopen this issue?

Might be option for forceful pull might be useful for rolling update without yml files.

Especially in kubectl set image deployment {} {image details}

@Gnouf

This comment has been minimized.

Copy link

Gnouf commented Aug 2, 2018

For Helm users :

helm upgrade --recreate-pods performs pods restart for the resource if applicable

@AndresPineros

This comment has been minimized.

Copy link

AndresPineros commented Aug 18, 2018

@Gnouf What does "if applicable" mean. Is this documented somewhere?

@br3ndonland br3ndonland referenced this issue Sep 2, 2018

Merged

Deploy to cloud and implement search #48

1 of 1 task complete
@kondaurovDev

This comment has been minimized.

Copy link

kondaurovDev commented Sep 12, 2018

Is it crazy idea to deploy watchtower on every node?
https://github.com/v2tec/watchtower

@dszmaj

This comment has been minimized.

Copy link

dszmaj commented Sep 12, 2018

@kondaurovDev if you are trying to manage containers your own way you are losing all benefits of kubernetes, a very bad idea, use helm, change annotations, env vars or image tags, your choice

@ObviousDWest

This comment has been minimized.

Copy link

ObviousDWest commented Sep 14, 2018

I am puzzled by the reluctance to accept use of tagged images as a good practice. If they are such a bad choice, why support their use in k8s at all? Answer: because they are useful and easy to understand. Given people find them useful and easy, why not provide a little better support for them? The big objection seems to be: but you can't tell what image was even used! So, why not expose that image's hash when getting pod info? If a user wants to know which image the automated update system pulled, they can trace things that way. To avoid down times (by deleting a service, or dropping replication to zero), why not make rolling-restart have a flag to re-pull the image? Or have a flag on apply. I think it is pretty awful to have to edit your deployment.yml file for every build just to increment a build number tag. And ide based iteration isn't so great at creating those unique numbers anyway. Think about what would increase the awesomeness of Kubernetes. What changes would make people gush and tell their friends? Look at my cool dev process! Compile; build image; tickle k8s; test; repeat. And leave the scripting and all-so-necessary precise change-control to DevOps and their super-tools. (I.e. support both approaches, please)

@dims

This comment has been minimized.

Copy link
Member

dims commented Sep 14, 2018

long-term-issue (note to myself)

@jlk

This comment has been minimized.

Copy link

jlk commented Sep 14, 2018

@kondaurovDev Keel might make more sense for k8s, altho I do like watchtower for stand-alone docker...

@deitch

This comment has been minimized.

Copy link

deitch commented Sep 25, 2018

FWIW, I went in the complete opposite direction. In all of my deployments, I now use not only tagged images, but include the sha256 hash on those images. Quite simply, I use latest almost nowhere, and even actual tags (even as far as using the git hash as a tag) almost nowhere, as I use the tag plus the sha256 of the manifest.

@spiddy

This comment has been minimized.

Copy link

spiddy commented Sep 25, 2018

@Flaviohnb be careful because this might fail you in the future, because it works only on some setups

If your replicas is greater than the number of maxunavailable https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#max-unavailable the first command will not scale down all pods with wrong image (assuming you use rollingupdate as deployment strategy which is the default) leaving you some pods with previous version.

also it might be important that during update you will not have any pod running to serve

@teddy-owen

This comment has been minimized.

Copy link

teddy-owen commented Oct 3, 2018

For those trying to automatically update your cluster to the latest images within a CI/CD pipeline, I recommend using kubectl to set the image to the latest digest as below. This works for me with Google Cloud Container Registry and Kubernetes Engine:

docker build -t docker/image/name .
docker push docker/image/name
kubectl set image deployment [deployment_name] [container_name]=$(docker inspect --format='{{index .RepoDigests 0}}' docker/image/name:latest)
@snobu

This comment has been minimized.

Copy link

snobu commented Nov 9, 2018

I am running into the same limitations. I agree that in an ideal setup every version would be explicitly tagged, but this can be cumbersome in highly automated environments. Think of dozens of containers with 100 new versions per day.

Also, when debugging and setting up a new infrastructure there are a lot of small tweaks made to the containers. Having a force-repull on a Deployment will make the process more frictionless.

^ This guy.

$ kubectl set image deployment monkey-business \
    monkey-business=some.shipya.rd/monkey-business --force-repull

Can we have this?

@ramnes

This comment has been minimized.

Copy link

ramnes commented Nov 16, 2018

I naturally expected kubectl apply --force to do exactly that. Bummed that it's not the case, and even more that there is no proper alternative.

@micw

This comment has been minimized.

Copy link

micw commented Nov 28, 2018

How about this Idea:
One could create an admission controller webhook that resolves tags to image hashes. So if a deployment with a tag is passed to the api-server, the tag is replaced by the image hash. If the same deployment is passed again to kubernetes and there's a new image, the hash changes and kubernetes would upgrade this delpoyment.
An additional check (e.g. pressence of a certain label) could be check to add this behaviour selectively.

@hiephm

This comment has been minimized.

Copy link

hiephm commented Dec 5, 2018

We can use Deployment to do that, according to Kubernetes Up and Running book:

Also, do not update the change-cause annotation when doing simple scaling operations. A modification of change-cause is a significant change to the template and will trigger a new rollout.

In this case, we can exploit this feature by updating kubernetes.io/change-cause annotations to force an update without changing image tag. I tried and it works.

@VanitySoft

This comment has been minimized.

Copy link

VanitySoft commented Dec 9, 2018

updating kubernetes.io/change-cause annotations to what allows this? @hiephm

@jb-sean

This comment has been minimized.

Copy link

jb-sean commented Dec 9, 2018

@hiephm @VanitySoft, this workaround has already been mentioned in the original issue’s list of workarounds, “Fake a change to the Deployment by changing something other than the image”. This includes changing/deleting/adding any annotation. There’s nothing special about the kubernetes.io/change-cause annotation in this case.

@ChristopherLClark

This comment has been minimized.

Copy link

ChristopherLClark commented Dec 12, 2018

kubectl apply -f file.yml --force should do this. But since it doesnt how do i set imagepull policy to always?

@kritishpahi

This comment has been minimized.

Copy link

kritishpahi commented Dec 13, 2018

@ChristopherLClark, setting --force doesn't work.
This actually works:
#27081 (comment)

FYI: Add
imagePullPolicy: Always
to your containers spec in deploment.yaml.

@jeliasson

This comment has been minimized.

Copy link

jeliasson commented Jan 13, 2019

Following the examples of apply a dynamic label, one could also apply the Git SHA to further identify deployments. I recommend envsubst for bringing in environment variables.

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: api
spec:
  template:
    metadata:
      labels:
        app: api
        sha: ${GIT_SHA_FULL}
    spec:
      containers:
      - name: api
        image: [...]:latest
@micw

This comment has been minimized.

Copy link

micw commented Jan 13, 2019

Be aware that you might to get random versions deployed this way (e.g. when a workload is migrated and the image tag has changed meanwhile). So using an admission controller webhook that resolves the tag to a hash would result in much more predictable results!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment