imagePullPolicy for the injected agent image fails admission control

[goffinf]

The following message is output when deploying a test application with the vault-k8s annotations:

Pods "vault-k8s-agent-webhook-demo-5b945c994b-g8xfn" is forbidden: spec.initContainers[0].imagePullPolicy: Unsupported value: "IfNotPresent": supported values: "Always"; Deployment does not have minimum availability.

We have admission controllers applied to the cluster in this order:

...,AlwaysPullImages,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,...

We have tried moving AlwaysPullImages to come AFTER MutatingAdmissionWebhook but that didn't help (same error).

Looking at injector-deployment.yaml I wonder whether it would be possible for you to expose the imagePullPolicy for the agent image as you do for the injector itself ? ..

      containers:
        - name: sidecar-injector
          {{ template "injector.resources" . }}
          image: "{{ .Values.injector.image.repository }}:{{ .Values.injector.image.tag }}"
          imagePullPolicy: "{{ .Values.injector.image.pullPolicy }}"
          env:
            - name: AGENT_INJECT_LISTEN
              value: ":8080"
            - name: AGENT_INJECT_LOG_LEVEL
              value: {{ .Values.injector.logLevel | default "info" }}
            - name: AGENT_INJECT_VAULT_ADDR
              value: {{ .Values.server.addr }}
            - name: AGENT_INJECT_VAULT_IMAGE
              value: "{{ .Values.injector.agentImage.repository }}:{{ .Values.injector.agentImage.tag }}"
            ...
            MAYBE SOMETHING LIKE THIS ???
            ...
            - name: AGENT_INJECT_VAULT_IMAGE_PULLPOLICY
              value: "{{ .Values.injector.agentImage.pullPolicy}}"

Regards

Fraser Goffin

[jasonodonnell]

Hi @goffinf, this is interesting because we aren't actually setting the pull policy for the init container: https://github.com/hashicorp/vault-k8s/blob/master/agent-inject/agent/container_init_sidecar.go#L57-L70. Regardless I think this is a good option to have.

As a work around, I think you can set .Values.injector.agentImage.tag to "latest", Kube will default to "Always" in this case.

[goffinf]

Hey Jason,

I can confirm that using the latest tag does default the imagePullPolicy to 'Always' and thus is a reasonable workaround for now. Clearly we want to remove that when this enhancement is implemented and merged.

[prune998]

Even make it configurable, default to Always or allow it to be changed using an annotation.
It seems so trivial, I can PR if you want.

Thanks

[SohamJ]

I can confirm that using the latest tag does default the imagePullPolicy to 'Always' and thus is a reasonable workaround for now. Clearly we want to remove that when this enhancement is implemented and merged.

@jasonodonnell , one of the use-cases of not using latest was to pin specific versions to know which versions were running in each of our kubernetes clusters/environments. That way, we could easily track the updates across our SDLC process. We also had an egde case where we had to compile and build a s390x arch agent to support our zLinux k8s workloads which had specific version associated to it.

Like @prune998 mentioned, this should be a change in the helm templates. I can take a stab at it if it's not in the works (and it seems there's no design consideration on limiting the pull policy based on the above discussion)

I however am not familar with Go to contribute to the agent-inject part in this repo (will still read through if I can)

[jghal]

We're having a related issue. We deployed a POD annotated for secrets and I see the init and agent containers when I describe the POD, but the POD is failing on ImagePullBackOff even though the image already exists on the node.

[benashz]

In v1.1.0 we added support for JSON-patch via annotations. You should now be able to specify the vault-agent ImagePullPolicy from the vault.hashicorp.com/agent-json-patch annotation.

Example:

spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-json-patch: '[{"op": "replace", "path": "/imagePullPolicy", "value": "IfNotPresent"}]'