Unable to pull pre-warm images from private docker registry

[pankaj-dahiya-devops]

I am unable to pull pre-warm image - "myregistry/action-nodejs-v10" from my private gitlab registry.

Steps to replicate:

1. Pull open whisk helm chart.
2. Edit values.yml and provide docker.registry.name, username, password:
3. Helm install

---

All images will be pulled successfully and all the pods will get into a running state. But Pre-warm pods will give an error as "ErrImagePull".

---

Events:
Type Reason Age From Message

---

Normal Scheduled 36s default-scheduler Successfully assigned default/wskpankaj-invoker-00-10-prewarm-nodejs10 to ip-172-31-39-89
Normal BackOff 13s (x2 over 35s) kubelet Back-off pulling image "registry.gitlab.com//openwhisk/action-nodejs-v10:1.16.0"
Warning Failed 13s (x2 over 35s) kubelet Error: ImagePullBackOff
Normal Pulling 1s (x3 over 35s) kubelet Pulling image "registry.gitlab.com/MYREP>/openwhisk/action-nodejs-v10:1.16.0"
Warning Failed 1s (x3 over 35s) kubelet Failed to pull image "registry.gitlab.com/MYREPO/openwhisk/action-nodejs-v10:1.16.0": rpc error: code = Unknown desc = failed to pull and unpack image "registry.gitlab.com/MYREPO/openwhisk/action-nodejs-v10:1.16.0": failed to resolve reference "registry.gitlab.com/MYREPO/openwhisk/action-nodejs-v10:1.16.0": failed to authorize: failed to fetch anonymous token: unexpected status: 403 Forbidden
Warning Failed 1s (x3 over 35s) kubelet Error: ErrImagePull

[dgrove-oss]

Is it only the preWarm pods that are having trouble pulling the image, or is it a problem for any pod being created by an invoker using this image to run user actions?

I'd speculate you are probably seeing the later, which means the imagePullSecret needs to be propagated into the pod template the invoker uses to create user action pods. There is poorly documented support for doing this by injecting a value into the invoker's configuration value: whisk.kubernetes.podTemplate https://github.com/apache/openwhisk/blob/master/core/invoker/src/main/resources/application.conf#L87-#L91.

[pankaj-dahiya-devops]

Hi, @dgrove-oss I am deploying Openwhisk via helm carts, In helm charts, there is an option in values.yaml which asks us to private docker registry and its credentials which are further passed to templates/_invoker-helpers.tpl where we define "openwhisk.docker_pull_runtimes" which includes these credentials.

So, as per me, providing registry credentials in values.yaml must be enough!

One more thing, as per industry standards, the User should provide a .Values.imagePullSecretsin values.yaml and that should be propagated to everywhere else but in Openwhisk this is completely useless as this .Values.imagePullSecrets is not passed anywhere to any pods of Openwhisk and we must provide our registry credentials to Openwhisk that too in plain text format!.

[dgrove-oss]

The helm chart supports using a private registry with the DockerContainerFactory. It is missing support for using the KubernetesContainerFactory with a private registry. Contributions would be welcome.

[pankaj-dahiya-devops]

Hi ... this thing should also be supported in KubernatesContainerFactory. Will raise a PR by next month for this. (Bit busy now). So closing the issue as of now.