Init containers not updated on upgrade

[Draiken]

Hi there!

I am using spec.initContainers on a deployment and during the first upgrade to add the initContainers, everything worked fine.
Second upgrade, I only changed things on the initContainers and the deployment didn't even get upgraded.

I tested changing something else and the deployment got updated but upon inspection, there were two annotations: pod.alpha.kubernetes.io/init-containers and pod.beta.kubernetes.io/init-containers, no spec.initContainers and the values on the annotations were still the ones from the first upgrade.

After that I attempted to use the annotations directly instead of the new spec field, and got the same results: the annotations never got updated

After several tests I've come to the conclusion that something is excluding initContainers from the upgrade diffs.

Am I doing something wrong here or is this a bug?
I have used initContainers on annotations before and they worked, but this was a few versions ago.

My helm is:

Client: &version.Version{SemVer:"v2.5.0", GitCommit:"012cb0ac1a1b2f888144ef5a67b8dab6c2d45be6", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.5.0", GitCommit:"012cb0ac1a1b2f888144ef5a67b8dab6c2d45be6", GitTreeState:"clean"}

And Kubernetes:

Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.6", GitCommit:"114f8911f9597be669a747ab72787e0bd74c9359", GitTreeState:"clean", BuildDate:"2017-03-28T13:54:20Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.4", GitCommit:"d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae", GitTreeState:"clean", BuildDate:"2017-05-19T18:33:17Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}

[jurgenweber]

This was posted in #helm-users and I made the comment that I have run into my initContainer details not updating before. I changed my templating to update the initspec and the annotations and it works after that.

...
  template:
    metadata:
      annotations:
        pod.alpha.kubernetes.io/init-containers: '[{"name":"code-copy","image":"example_image:latest","command":["cp","-r","/var/www/html/","/html"],"resources":{},"volumeMounts":[{"name":"app-volume","mountPath":"/html"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}]'
        pod.beta.kubernetes.io/init-containers: '[{"name":"code-copy","image":"example_image:latest","command":["cp","-r","/var/www/html/","/html"],"resources":{},"volumeMounts":[{"name":"app-volume","mountPath":"/html"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}]'
      labels:
        app: hip-new
        hip-new: www
      name: www
      namespace: engineering
    spec:
      initContainers:
        - name: code-copy
          image: example_image:latest
          command: ["cp", "-r", "/var/www/html/", "/html"]
          volumeMounts:
            - name: app-volume
              mountPath: /html
      containers:
        - name: fpm
...

sort of idea, so if you want to update the image, since you have a later version.. change the latest in all 3 places to what you need.

Then it works, I did note you saying you knew about this and it is still not working. I dunno.

[Draiken]

I use a variable {{ .Values.image.tag }} matching on the init container and regular containers for the docker image tag. When upgrading, the regular containers get updated and the initContainers do not.

...
spec:
  template:
    spec:
      initContainers:
        - name: assets
          image: "{{ .Values.assets.repository }}:{{ .Values.image.tag }}"
      containers
        - name: other-container
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
...

I will try to replicate the init container on both annotations and the spec to see what happens...

[Draiken]

Just to clarify what I tried so far:

• Using only alpha init container annotation
• Using only spec.initContainers

I have yet to try:

• Use only the beta annotation
• Use both the beta and alpha annotations
• Use the both annotations and the spec.initContainers

[Draiken]

I managed to replicate the issue on a new chart:

helm create init-bug
echo '      initContainers: [ { name: "sleeper", image: "alpine:{{ .Values.alpineVersion }}", args: ["/bin/ash", "-c", "sleep 10"] } ]' >> init-bug/templates/deployment.yaml
helm install init-bug --name buggy --set alpineVersion=3.5
helm upgrade buggy init-bug --set alpineVersion=3.6
kubectl get deployment buggy-init-bug -o jsonpath="{.spec.template.spec.initContainers[0].image}"

Running this will output

alpine:3.5

If I run the upgrade on --dry-run and --debug the init container looks correct, but when actually running the code, the deployment is not even changed.

[miadabrin]

Same here. Init containers are not changed on kubectl apply -f to the same resource file

[Draiken]

This is probably related to kubernetes/kubernetes#47264
I'll try changing the beta annotation. Hopefully it'll work

[jstriebel]

Same here when using spec.initContainers in a CronJob. Changes are not reflected in Kubernetes when upgrading. Requesting to change the label of this issue to Bug.

Example:

apiVersion: batch/v2alpha1
kind: CronJob
spec:
  jobTemplate:
    spec:
      template:
        spec:
          initContainers:
          …

[vincbe]

Same problem here.
I got an init-container and at least two containers ,
both the init-container and container 1 are set to use the same image defined by parameters:
image: {{ .Values.Image }}:{{ .Values.ImageVersion }}

So when I do update ".Values.ImageVersion" with a new version and do a help upgrade on the chart, the container 1 does use a new version but not the init-container!!

That's a real problem, but seems to be a problem with Kubernetes more than Helm

[vincbe]

Having the init container definition in double with the beta annotation is a work around that works for me to wait Kubernetes 1.8:
kubernetes/kubernetes#47264 (comment)

[Draiken]

Yes, this is what worked for me too. I have the same init container as a beta annotation and as initContainer on the spec and it works. As stated it's a workaround but I don't feel this will get fixed so quickly

[bacongobbler]

Sounds like this is an upstream issue which we have no control over, unfortunately. I'd suggest following up at kubernetes/kubernetes if there are further issues with updating init containers. Sorry we can't help you with this!

[jrnt30]

As a newer comment on the thread, it looks like explicitly setting the annotations to null will also have the desired effect of only looking at the spec.