Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Force pods to re-pull an image without changing the image tag #33664

Closed
yissachar opened this issue Sep 28, 2016 · 105 comments
Closed

Force pods to re-pull an image without changing the image tag #33664

yissachar opened this issue Sep 28, 2016 · 105 comments

Comments

@yissachar
Copy link

@yissachar yissachar commented Sep 28, 2016

Problem

A frequent question that comes up on Slack and Stack Overflow is how to trigger an update to a Deployment/RS/RC when the image tag hasn't changed but the underlying image has.

Consider:

  1. There is an existing Deployment with image foo:latest
  2. User builds a new image foo:latest
  3. User pushes foo:latest to their registry
  4. User wants to do something here to tell the Deployment to pull the new image and do a rolling-update of existing pods

The problem is that there is no existing Kubernetes mechanism which properly covers this.

Current Workarounds

  • Always change the image tag when deploying a new version
  • Refer to the image hash instead of tag, e.g. localhost:5000/andy/busybox@sha256:2aac5e7514fbc77125bd315abe9e7b0257db05fe498af01a58e239ebaccf82a8
  • Use latest tag or imagePullPolicy: Always and delete the pods. New pods will pull the new image. This approach doesn't do a rolling update and will result in downtime.
  • Fake a change to the Deployment by changing something other than the image

Possible Solutions

  • #13488 If rolling restart were implemented, users could do a rolling-restart to pull the new image.
  • Have a controller that watches the image registry and automatically updates the Deployment to use the latest image hash for a given tag. See #1697 (comment)

cc @justinsb

@justinsb
Copy link
Member

@justinsb justinsb commented Sep 28, 2016

This is indeed important, and I think there are two cases:

  1. when we have a full CI system like Jenkins (aka “do I really have to use sed”)
  2. we have a limited system like dockerhub that only re-tags latest

@yujuhong
Copy link
Member

@yujuhong yujuhong commented Oct 14, 2016

@yissachar using :latest tag IMO is not the best practice as it's hard to track what image is really in use in your pod. I think tagging images by versions or using the digests is strictly better than reusing the same tag. Is it really such a hassle to do that?

@Arachnid
Copy link

@Arachnid Arachnid commented Oct 14, 2016

@yujuhong Sometimes it's very useful to be able to do this. For instance, we run a testing cluster that should run a build from the latest commit on the master branch of our repository. There aren't tags or branches for every commit, so ':latest' is the logical and most practical name for it.

Wouldn't it make more sense if Kubernetes stored and checked the hash of the deployed container instead of its (mutable) name anyway, though?

@yissachar
Copy link
Author

@yissachar yissachar commented Oct 14, 2016

@yujuhong I agree that if you can do so then you should (and I do!). But this question comes up quite frequently and often users cannot easily tag every build (this often arises with CI systems). They need a solution with less friction to their process, and this means they want to see some way of updating a Deployment without changing the image tag.

@dominiek
Copy link

@dominiek dominiek commented Oct 14, 2016

I am running into the same limitations. I agree that in an ideal setup every version would be explicitly tagged, but this can be cumbersome in highly automated environments. Think of dozens of containers with 100 new versions per day.

Also, when debugging and setting up a new infrastructure there are a lot of small tweaks made to the containers. Having a force-repull on a Deployment will make the process more frictionless.

@yujuhong
Copy link
Member

@yujuhong yujuhong commented Oct 14, 2016

I am running into the same limitations. I agree that in an ideal setup every version would be explicitly tagged, but this can be cumbersome in highly automated environments. Think of dozens of containers with 100 new versions per day.

Hmm....I still think automatically tagging images by commit hash would be ideal, but I see that it may be difficult to do for some CI systems.

In order to do this, we'd need (1) a component to detect the change and (2) a mechanism to restart the pod.

Also, when debugging and setting up a new infrastructure there are a lot of small tweaks made to the containers. Having a force-repull on a Deployment will make the process more frictionless.

This sounds reasonable.

/cc @pwittrock, who has more context on the CI systems.

@Arachnid
Copy link

@Arachnid Arachnid commented Oct 14, 2016

Hmm....I still think automatically tagging images by commit hash would be ideal, but I see that it may be difficult to do for some CI systems.

Creating a tag for every single commit is also pretty pointless - commits already have unique identifiers - especially when you only care about the last one.

What I don't understand is why Kubernetes treats tags as if they're immutable, when they're explicitly mutable human-readable names for immutable identifiers (the hash of the manifest).

@pwittrock
Copy link
Member

@pwittrock pwittrock commented Oct 15, 2016

@erictune @janetkuo

This could live either outside the deployment in a CICD system that forces forces a new deployment rollout. Alternatively, it could be a field on the deployment. WDYT?

@alphashuro
Copy link

@alphashuro alphashuro commented Oct 18, 2016

What is the consensus on this?

@pwittrock
Copy link
Member

@pwittrock pwittrock commented Oct 18, 2016

Longer term some CICD system should support this.

Immediate term: It would probably be simple to create a controller that listens for changes to a container registry and then updates a label on all deployments with a specific annotation. You could install this controller into your kubernetes cluster using helm.

I will try to hack a prototype together later this week.

@pwittrock
Copy link
Member

@pwittrock pwittrock commented Oct 26, 2016

Quick question - why not set an annotation on the pod template with the current time to force the repull. I believe this would execute an update using the deployment's strategy to rollout the new im age.

I put together an example of how to write a controller to do this in response to webhook callbacks from dockerhub. I need to add some documentation and then will post the example here. Ideally I would put together a helm chart for this as well.

@pwittrock
Copy link
Member

@pwittrock pwittrock commented Oct 26, 2016

FYI here is a simple controller for pushing deployment updates in response to webhook callbacks. It natively supports dockerhub, but you can manually post to it from the command line.

container-image-pusher

@brendandburns
Copy link
Contributor

@brendandburns brendandburns commented Nov 4, 2016

fwiw, I don't think we should support this as a proper Kubernetes API.

I don't quite understand the use case, as I see it there are two scenarios:

  1. you are using 'latest' for testing (this is the "no sed" use case), in this case, downtime is fine, and indeed the right approach is likely to completely blow away your stack and redeploy from scratch to get a clean run.

  2. you are running 'latest' in prod. If you do this, you are just asking for trouble, there are a myriad different failure modes that can occur (and perhaps more importantly) if you aren't tagging your images with the git hash you don't have a rollback path (since the old latest has been blown away by the newer latest)

Finally, I believe that kubectl set <...> eliminates the need for sed

@justinclayton or @yissachar is there a use case I'm missing here?

@Arachnid
Copy link

@Arachnid Arachnid commented Nov 4, 2016

  1. you are using 'latest' for testing (this is the "no sed" use case), in this case, downtime is fine, and indeed the right approach is likely to completely blow away your stack and redeploy from scratch to get a clean run.

I'm not sure I follow the argument here. Downtime isn't fine in our use-case, of running monitoring nodes of the latest instances of our software. It seems sensible to be able to apply the same deployment mechanics to this as to anything else.

More broadly, docker tags, like most name services, are fundamentally mutable by design - and docker repos provide a way to resolve a tag to the current image hash. I don't understand why Kubernetes associates the mutable tag with a deployed pod, then treats it as immutable, instead of just using the immutable identifier in the first place.

Finally, I believe that kubectl set <...> eliminates the need for sed

Perhaps, but it still leaves the task of resolving tag name to image hash up to the user, something that's definitely nontrivial to do with existing tools.

@serverhorror
Copy link

@serverhorror serverhorror commented Nov 4, 2016

@brendandburns I'm interested in this as well. Not for the reasons of updating the pods.

My situation is this: Pods and Containers are pretty stable but the data moves way faster. Our data sets span 100s of GBs per file with 100s of files (genomic data, life sciences). And since a lot of the software is academic there isn't much engineering effort going into it. Currently the easiest way to "redeploy" is to replace a config map that points to the new data sets. Kubernetes takes care of replacing the actual config file in the container but right now there's no way to trigger a a kind of rolling-update so that pods get killed and restarted the same way it would happen with an update to the actual container versions. I don't want to get into the business of image management too much so I try not to update images every time data changes.

Does that makes sense?

I'm happy to go any other path, but my current experience is that this seems to be the way to go when there's not enough development bandwidth to fix the underlying issues.

@kargakis
Copy link
Member

@kargakis kargakis commented Nov 4, 2016

#13488 seems related

@brendandburns
Copy link
Contributor

@brendandburns brendandburns commented Nov 6, 2016

@serverhorror I think the way that I would accomplish what you want is that I would set up a side car container that is in the same pod as your main container. The job of that sidecar is to monitor the config file and send a signal (e.g. SIGHUP or SIGKILL) to your main container that indicates that the data file has changed.

You could also use container health checks e.g. set up a health check for your 'main' container to point to a web service hosted by your sidecar. Whenever the sidecar changes, the health check goes 'unhealthy' and the kubelet will automatically restart your main container.

@Arachnid I guess I fundamentally believe that tags should not be used in a mutable matter. If you use image tags in a mutable way, then the definition stops having meaning, you no longer can know for sure what is running in a particular container just by looking at the API object. Docker may allow you to mutate tags on images, but I think that the Kubernetes philosophy (and hard-won experience of running containerized systems at scale) is that mutable tags (and 'latest' in particular) are very dangerous to use in a production environment.

I agree that the right thing to do is to apply the same deployment mechanics in test and in prod, given that, and the belief that latest is dangerous in production, it means the right answer is to use git-sha for your tag and use the Deployment object to do rollouts for both test and prod.

Here are some examples of concrete production issues that I ran into due to the use of latest:

  • A task restart in the middle of the night accidentally 'upgraded' one of my apps to a debug (e.g. slow) build b/c someone mistakenly moved the 'latest' tag.
  • I believed that a server was actually fully upgraded because it was running 'latest' but actually the image pull on the machine was perma-failing and so it was lagging behind on an old version.
  • Auto-scaling caused an accidental 'upgrade' because when it scaled up via 'latest' it created containers using the new image, when I wasn't ready to roll it out yet.

I hope that helps explain why I think that mutable labels are a dangerous idea.

@Arachnid
Copy link

@Arachnid Arachnid commented Nov 6, 2016

I guess I fundamentally believe that tags should not be used in a mutable matter. If you use image tags in a mutable way, then the definition stops having meaning, you no longer can know for sure what is running in a particular container just by looking at the API object.

Agreed, as-is they're dangerous, but this could be trivially resolved by having the API object retain the hash of the container image as the permanent identifier for it, rather than assuming the (mutable) tag won't change. This seems like a fundamental mismatch between how Docker treats tags and how Kubernetes treats them, but it seems resolvable, to me. Every one of the problems you list below could be resolved by storing and explicitly displaying the hash of the currently running container.

Tagging images by their git hashes doesn't really express what I mean when I create a deployment, and introduces awkward dependencies requiring me to propagate those tags through the system.

@serverhorror
Copy link

@serverhorror serverhorror commented Nov 8, 2016

@brendandburns Right, liveness checks seem to be another easy way. That is serving my needs, could have thought of that. Consider my argument for this taken back :)

@deitch
Copy link
Contributor

@deitch deitch commented Nov 29, 2016

@brendandburns and @yujuhong: I could see this being useful in a number of use cases, where "latest" is used in prod.

"latest" is dangerous in production

Depends on how "latest" gets used. I have worked with a number of environments where there is a single image registry that supports prod/testing/etc. (which makes sense). However, the given repos can be populated only by CI. Builds off of any branch get tagged correctly with versions, but builds off HEAD from master (which pass all tests of course) also get tagged "latest".

Prod environments, in turn, point at "latest". That way I don't need to update anything about versions for prod; l just need to say, "go rolling update" (either automatically or when a human approves, which hopefully will be removed from the process very soon).

To answer the "danger" question:

  1. No human can ever mistakenly tag something latest because humans do not get to push to the repos in question.
  2. I always know what version I am running by looking at image tags.
  3. I am much more worried about letting a human update my deployment config and entering tag "1.2.3" instead of "1.2.4" (or worse "145abcd6" instead of "145adcd6") and deploying. Which is why we went "latest" in the first place.

So:

  1. I think "rolling update of a deployment without changing tags" is a real use case that real people have, worth supporting.
  2. I am more than happy to switch to immutable tags (and avoid the problem), if I can find a way to not involve humans in the tag-change (step 3) process above.

I guess I could whip up a script or Web app that lists all available tags that come from "master" and makes them pick one, and when we go full automated, have the CI also pull the deployment, update the image, and redeploy?

@alirezaDavid
Copy link

@alirezaDavid alirezaDavid commented Jan 9, 2017

if kubectl apply -f for-god-sake-update-latest-image.yaml can update latest image we should really happy. (with ImagePullPollicy: Always)

@micw
Copy link

@micw micw commented Feb 12, 2019

@jeliasson If you are able to resolve the tag to a hash, It'sprobably a better Idea to use the has has image tag and annotate with the original tag. This way you will ensure that all instances run with the same version if your tags are mutable and that the version is only changed if you redeploy.
As sayed above, an Admission Web Hook should be able to do this tag to sha resolution very well because there's everything in place (image tag name, pull secrets).

@jmcdonagh
Copy link

@jmcdonagh jmcdonagh commented Feb 13, 2019

Reading more into it I see why it's considered a bad practice. Bunch of cases where different versions could be deployed accidentally... for small shops like ours I don't think those cases are very frequent.

@philpep
Copy link

@philpep philpep commented Mar 9, 2019

Hi,

My usecase is to deploy security updates automatically, in this case the image tag doesn't change while target image is updated. I wrote a tool to ensure my pods are running on the latest digest, also this avoid having some nodes not running exactly the same image.

The code is available at https://github.com/philpep/imago

The main idea is to update deployments/daemonset to use images specifiers with sha256 digest, while original configuration is stored on an annotation.

Features:

  • Using ImagePullSecrets to check latest digest on registries
  • Automatically update Deployments and DaemonSet
  • Can run outside of the cluster
  • Can run inside the cluster (I use this in a daily CronJob)
  • Can target specific deployments, daemonset, namespaces etc
  • Has a mode for checking running pods (a bit slower but less "intrusive" on updates proposals)

My plans for future enhancements are:

  • Think about writing this as an AdmissionController modifying images on the fly on submission
  • http webhook mode (this could allow CI to trigger deployments without needing direct access to the cluster)

Let me known what you think of this tool, It's still experimental but for my usecase it just work fine :)

@micw
Copy link

@micw micw commented Mar 9, 2019

Hello @philpep ,
this is great stuff! On plain docker there was a tool called "watchtower" doing similar. Up to now I did not find a k8s counterpart. I'm very happy that we do have a similar tool for k8s now.
The admission controller would be an excelent extension to it because it would ensure that when one deploys a mutable tag, that the whole cluster runs the same version/hash of it. It also would ensure that the changes made by imago would not be overwritten on the next deployment.

I'll try to do a test install on one of my clusters in the next 1-2 weeks and give you more qualified feedback ;-)

Best regards,
Michael

@bgrant0607
Copy link
Member

@bgrant0607 bgrant0607 commented Mar 27, 2019

Yes, we recommend pulling by digest. See also #1697.

@jgirdner
Copy link

@jgirdner jgirdner commented Mar 31, 2019

64471948

Editing deploy scripts with the digest doesn't sound safe either. Just saying. Humans are humans. We make mistakes and could accidentally use the wrong digest.

@CodeSwimBikeRunner
Copy link

@CodeSwimBikeRunner CodeSwimBikeRunner commented Apr 8, 2019

@jgirdner it should be your build server doing it, and you should use replacement characters. it is a very common ci/cd process.

@frbaron
Copy link

@frbaron frbaron commented May 30, 2019

Sharing our approach in CD (who is managing versions/images/tags)

Our deploy-xxx stage is basically:

  • kubectl patch deployment $IMAGE_NAME -p "{"spec":{"template":{"metadata":{"labels":{"date":"date +'%s'"}}}}}"
  • kubectl rollout status deployment $IMAGE_NAME -w --request-timeout='600s'

We change a date field from the pipeline, check rollout status which returns as soon as our service is Running 1/1.

And, no one messes with Kubernetes template at every software change. The deployment describes where the image is, based on :dev, :int, :prd tag managed by the CD once test-xxx stages are ok).

In short, very surprised that kubectl does not have a simple "restart" action. Aliasing just the date is the closest we got to that ;-)

@jgirdner
Copy link

@jgirdner jgirdner commented May 30, 2019

We ended up building a simple python script that builds our yaml files. Here is an example of that.

import yaml
import subprocess

def yaml_dump(filepath, data):
    with open(filepath, "w") as file_descriptor:
        yaml.dump(data, file_descriptor)

def get_git_revision_hash(container):
    image = f'gcr.io/xxxxx/{container}'
    return subprocess.check_output(['gcloud', 'container', 'images', 'describe', image]).decode('ascii').strip().split('fully_qualified_digest: ')[1].split('\n')[0]

if __name__=='__main__':

    ## Generate Beat Yaml
    beatimage = get_git_revision_hash('celery-beat')
    beatfilepath = "prod-beat.yaml"
    beatdata = {
        "apiVersion": "extensions/v1beta1",
        "kind": "Deployment",
        "metadata": {
            "creationTimestamp": None,
            "labels": {
                "io.kompose.service": "celery-beat"
            },
            "name": "celery-beat"
        },
        "spec": {
            "replicas": 1,
            "strategy": {
                "type": "RollingUpdate",
                "rollingUpdate": {
                    "maxSurge": 1,
                    "maxUnavailable": 0
                }
            },
            "minReadySeconds": 5,
            "template": {
                "metadata": {
                    "creationTimestamp": None,
                    "labels": {
                        "io.kompose.service": "celery-beat"
                    }
                },
                "spec": {
                    "containers": [
                        {
                            "env": [
                                {
                                    "name": "C_FORCE_ROOT",
                                    "value": "'true'"
                                },
                                {
                                    "name": "GOOGLE_APPLICATION_CREDENTIALS",
                                    "value": "certs/gcp.json"
                                },
                                {
                                    "name": "XXXXXX_ENV",
                                    "value": "prod"
                                }
                            ],
                            "image": beatimage,
                            "name": "XXXXXX-celery-beat",
                            "resources": {
                                "requests": {
                                    "memory": "200Mi",
                                    "cpu": "150m"
                                },
                                "limits": {
                                    "memory": "300Mi",
                                    "cpu": "200m"
                                }
                            }
                        }
                    ],
                    "restartPolicy": "Always"
                }
            }
        },
        "status": {}
    }
    yaml_dump(beatfilepath, beatdata)


    ## Generate Celery Yaml
    celeryimage = get_git_revision_hash('celery-worker')
    celeryfilepath = "prod-celery.yaml"
    celerydata = {
        "apiVersion": "extensions/v1beta1",
        "kind": "Deployment",
        "metadata": {
            "creationTimestamp": None,
            "labels": {
                "io.kompose.service": "celery-worker-1"
            },
            "name": "celery-worker-1"
        },
        "spec": {
            "replicas": 3,
            "strategy": {
                "type": "RollingUpdate",
                "rollingUpdate": {
                    "maxSurge": 1,
                    "maxUnavailable": 0
                }
            },
            "minReadySeconds": 5,
            "template": {
                "metadata": {
                    "creationTimestamp": None,
                    "labels": {
                        "io.kompose.service": "celery-worker-1"
                    }
                },
                "spec": {
                    "containers": [
                        {
                            "env": [
                                {
                                    "name": "CELERY_NAME",
                                    "value": "celery-pods"
                                },
                                {
                                    "name": "GOOGLE_APPLICATION_CREDENTIALS",
                                    "value": "certs/gcp.json"
                                },
                                {
                                    "name": "XXXXXX_ENV",
                                    "value": "prod"
                                }
                            ],
                            "image": celeryimage,
                            "name": "XXXXXX-celery-worker-1",
                            "resources": {
                                "requests": {
                                    "memory": "500Mi",
                                    "cpu": "500m"
                                },
                                "limits": {
                                    "memory": "600Mi",
                                    "cpu": "600m"
                                }
                            }
                        }
                    ],
                    "restartPolicy": "Always",
                    "terminationGracePeriodSeconds": 60
                }
            }
        },
        "status": {}
    }
    yaml_dump(celeryfilepath, celerydata)


    ## Generate Uwsgi Yaml
    uwsgiimage = get_git_revision_hash('uwsgi')
    uwsgifilepath = "prod-uwsgi.yaml"
    uwsgidata = {
        "apiVersion": "extensions/v1beta1",
        "kind": "Deployment",
        "metadata": {
            "creationTimestamp": None,
            "labels": {
                "io.kompose.service": "uwsgi"
            },
            "name": "uwsgi"
        },
        "spec": {
            "replicas": 3,
            "strategy": {
                "type": "RollingUpdate",
                "rollingUpdate": {
                    "maxSurge": 1,
                    "maxUnavailable": 0
                }
            },
            "minReadySeconds": 5,
            "template": {
                "metadata": {
                    "labels": {
                        "io.kompose.service": "uwsgi"
                    }
                },
                "spec": {
                    "containers": [
                        {
                            "env": [
                                {
                                    "name": "GOOGLE_APPLICATION_CREDENTIALS",
                                    "value": "certs/gcp.json"
                                },
                                {
                                    "name": "XXXXXX_ENV",
                                    "value": "prod"
                                }
                            ],
                            "image": uwsgiimage,
                            "name": "XXXXXX-uwsgi",
                            "ports": [
                                {
                                    "containerPort": 9040
                                }
                            ],
                            "readinessProbe": {
                                "httpGet": {
                                    "path": "/health/",
                                    "port": 9040
                                },
                                "initialDelaySeconds": 5,
                                "timeoutSeconds": 1,
                                "periodSeconds": 15
                            },
                            "livenessProbe": {
                                "httpGet": {
                                    "path": "/health/",
                                    "port": 9040
                                },
                                "initialDelaySeconds": 60,
                                "timeoutSeconds": 1,
                                "periodSeconds": 15
                            },
                            "resources": {
                                "requests": {
                                    "memory": "1000Mi",
                                    "cpu": "1800m"
                                },
                                "limits": {
                                    "memory": "1200Mi",
                                    "cpu": "2000m"
                                }
                            }
                        }
                    ],
                    "hostname": "uwsgi",
                    "restartPolicy": "Always",
                    "terminationGracePeriodSeconds": 60
                }
            }
        },
        "status": {}
    }
    yaml_dump(uwsgifilepath, uwsgidata)

    ## Generate Flower Yaml
    flowerimage = get_git_revision_hash('celery-flower')
    flowerfilepath = "prod-flower.yaml"
    flowerdata = {
        "apiVersion": "extensions/v1beta1",
        "kind": "Deployment",
        "metadata": {
            "creationTimestamp": None,
            "labels": {
                "io.kompose.service": "celery-flower"
            },
            "name": "celery-flower"
        },
        "spec": {
            "replicas": 1,
            "strategy": {
                "type": "RollingUpdate",
                "rollingUpdate": {
                    "maxSurge": 1,
                    "maxUnavailable": 0
                }
            },
            "minReadySeconds": 5,
            "template": {
                "metadata": {
                    "creationTimestamp": None,
                    "labels": {
                        "io.kompose.service": "celery-flower"
                    }
                },
                "spec": {
                    "containers": [
                        {
                            "env": [
                                {
                                    "name": "GOOGLE_APPLICATION_CREDENTIALS",
                                    "value": "certs/gcp.json"
                                },
                                {
                                    "name": "XXXXXX_ENV",
                                    "value": "prod"
                                }
                            ],
                            "image": flowerimage,
                            "name": "XXXXXX-celery-flower",
                            "ports": [
                                {
                                    "containerPort": 5555
                                }
                            ],
                            "resources": {
                                "requests": {
                                    "memory": "200Mi",
                                    "cpu": "400m"
                                },
                                "limits": {
                                    "memory": "300Mi",
                                    "cpu": "600m"
                                }
                            }
                        }
                    ],
                    "hostname": "flower",
                    "restartPolicy": "Always"
                }
            }
        },
        "status": {}
    }
    yaml_dump(flowerfilepath, flowerdata)

@ramnes
Copy link

@ramnes ramnes commented May 30, 2019

Guys, Kubernetes 1.15 will ship with a kubectl rollout restart command. See #13488.

@ObviousDWest
Copy link

@ObviousDWest ObviousDWest commented May 31, 2019

@ramnes
Copy link

@ramnes ramnes commented May 31, 2019

But I still want it for statefulSets. Maybe Deployments need the option of controlling statefulSet, not just replicaSet.

kubectl rollout restart also works with StatefulSets and DaemonSets. 😉

@cppforlife
Copy link

@cppforlife cppforlife commented Dec 19, 2019

i just ended up on this issue from following some blog posts, etc.

one solution as mentioned above was to...

Refer to the image hash instead of tag, e.g. localhost:5000/andy/busybox@sha256:2aac5e7514fbc77125bd315abe9e7b0257db05fe498af01a58e239ebaccf82a8

for folks who want to do it in an automated way, we ended writing some time ago a little tool, kbld (https://get-kbld.io), that transform image references to their digest equivalents. even though we did this to lock down which image is being used, it would also solve this problem as well in a more automated manner.

@chetandev
Copy link

@chetandev chetandev commented Feb 18, 2020

When hpa enabled, kubectl rollout restart creates max number of pods

@dapseen
Copy link

@dapseen dapseen commented Apr 12, 2020

Workaround for this is to implementing SHA digest , which is really working for me

@VanitySoft
Copy link

@VanitySoft VanitySoft commented Apr 12, 2020

@sean-krail
Copy link

@sean-krail sean-krail commented Apr 13, 2020

@VanitySoft, @dapseen is saying to pull by Docker image SHA digest instead of by tag names. This would be a change in your CI/CD workflow. You'd have to add something like this (assuming you're using Docker Hub):

docker_token=$(curl -s -u "${DOCKER_USERNAME}:${DOCKER_PASSWORD}" -H "Accept: application/vnd.docker.distribution.manifest.v2+json" "https://auth.docker.io/token?service=registry.docker.io&scope=repository:${DOCKER_REPO}:pull&account=${DOCKER_USERNAME}" | jq -r '.token')
docker_digest=$(curl -s -I -H "Authorization: Bearer ${docker_token}" -H "Accept: application/vnd.docker.distribution.manifest.v2+json" "https://index.docker.io/v2/${DOCKER_REPO}/manifests/${DOCKER_TAG}" | grep 'Docker-Content-Digest' | cut -d' ' -f2 | tr -d '\r')
unset docker_token

Then the image is referenced as ${DOCKER_REPO}@${docker_digest} instead of ${DOCKER_REPO}:${DOCKER_TAG}.

This is the only way to achieve true idempotent deployments since digests don't change.

@arjunkrishnasb
Copy link

@arjunkrishnasb arjunkrishnasb commented May 25, 2020

Hi,
I have a very similar use case where i cannot change the image tag every time and still the pods should be recreated if the image is changed in the image registry. I did see about kubectl rollback restart How to do the same with helm?
My deployment image in helm templates is pointing to latest and so even if i make any changes to the image and push to the registry, the changes are not reflected and the pods are not recreated in the deployment.

Any help would be really appreciated.

Thanks

@dkapanidis
Copy link

@dkapanidis dkapanidis commented May 25, 2020

The kubectl rollback restart command applies an annotation timestamp to the pod Spec that forces it to mutate and is what activates the rollout of the new Pod (note you have to have an imagePullPolicy: Always so that the fresh Pod actually pulls the image)

spec:
  template:
    metadata:
      annotations:
        kubectl.kubernetes.io/restartedAt: "2020-05-25T19:13:21+02:00"

In order to replicate that in helm you can use the same pattern. We use a value flag to activate a force restart when needed by adding to the deployment the following line:

spec:
  template:
    metadata:
      annotations:
        {{if .Values.forceRestart }}helm.sh/deploy-date: "{{ .Release.Time.Seconds }}"{{end}}

if forceRestart is true then the helm template engine adds annotation with the current time.

@arjunkrishnasb
Copy link

@arjunkrishnasb arjunkrishnasb commented May 25, 2020

Hey dkapanidis,
Thanks for the quick response. In my case this solution might not work. Because we are deploying our apps though CI/CD. So i need the pods to be recreated only if there is a change in the image in image registry else they can stay the same. If i am using the above solution, i need to manually add forceRestart flag everytime i make any changes to the image and thus breaks the purpose of CI/CD automation. Also the application should not have any downtime

@dkapanidis
Copy link

@dkapanidis dkapanidis commented May 25, 2020

We use this on CI/CD and depending on the branch the CICD builds if it is the "develop" branch (or "master" depending on your git flow) that translates to "latest" tags in docker registry then the CICD activates the flag (so that it is only used when the image is overwritten, not during tag releases). I'm assuming here that the CI and CD are triggered together and everytime an image is build, the deployment is also done, which means you always need to redeploy on those cases.

As long as the downtime, there should be none as the rollout of the deployment takes care of that.

@philpep
Copy link

@philpep philpep commented May 25, 2020

Hi @Arjunkrisha , I built a tool for this: https://github.com/philpep/imago where you can just use invoke withimago -restart. It check running pod image sha256 digest and compare with the registry. In case they don't match it add an annotation to the deployment/statefulset/daemonset to trigger a restart (assuming imagePullPolicy is "Always").

@arjunkrishnasb
Copy link

@arjunkrishnasb arjunkrishnasb commented May 26, 2020

Hi philpep,

  • Great Job on the tool that you made. I will definitely try to test it out more and get back to you in case of any feedback.
  • But sounds like there is no native kubernetes way or helm way to achieve this yet. I am not sure if someone is looking into this use case. If anyone knows that this is already tracked in kubernetes or helm then please let me know the issue number, else i believe i should create one.

Thanks!

@arjunkrishnasb
Copy link

@arjunkrishnasb arjunkrishnasb commented May 26, 2020

We use this on CI/CD and depending on the branch the CICD builds if it is the "develop" branch (or "master" depending on your git flow) that translates to "latest" tags in docker registry then the CICD activates the flag (so that it is only used when the image is overwritten, not during tag releases). I'm assuming here that the CI and CD are triggered together and everytime an image is build, the deployment is also done, which means you always need to redeploy on those cases.

As long as the downtime, there should be none as the rollout of the deployment takes care of that.

I like this idea as well. Will check out and let you know which works best

@guieckert
Copy link

@guieckert guieckert commented Feb 26, 2021

Hi.

I know this case is closed, but here is my solution and why it works.

In the Kubernetes documentation:

"This means that the new revision is created if and only if the Deployment's Pod template (.spec.template) is changed, for example if you update the labels or container images of the template."

[Kubernetes](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#rolling-back-a-deployment:~:text=This%20means%20that%20the%20new%20revision,or%20container%20images%20of % 20the% 20template)

I have Gitlab to send to Kubernetes, I have access to CI_COMMIT_SHA variable to differentiate between commit and commit.
I believe it is the same if you inform the date and time.

deploy.yaml

spec:
    template:
     metadata:
       annotations:
         commit_sha: CI_COMMIT_SHA

Using the "apply" command it will perform the update because the annotation has changed.

kubectl apply -f deploy.yaml

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet