Specify wait container image in Argo/KubeFlow pipeline

[saggar]

Hi. We're facing Docker rate limiting with our KubeFlow pipelines on AWS EKS. Is there a way for users to specify the Argo wait container image (here argoproj/argoexec:v2.3.0) when submitting a pipeline step? This is something a user can do for the main container of the pod. However, couldn't find a similar option for the wait container. Thanks!

kind: EventList
apiVersion: v1
metadata:
  selfLink: /api/v1/namespaces/kubeflow-admin/events
  resourceVersion: '58738437'
items:
# …
# …
 
    involvedObject:
      kind: Pod
      namespace: kubeflow-admin
      name: multi-z5568-154393337
      uid: 6cf5c442-72ae-4fe2-b891-28a37ad70f65
      apiVersion: v1
      resourceVersion: '58717425'
      fieldPath: 'spec.containers{wait}'
    reason: Failed
    message: >-
      Failed to pull image "argoproj/argoexec:v2.3.0": rpc error: code = Unknown
      desc = Error response from daemon: toomanyrequests: You have reached your
      pull rate limit. You may increase the limit by authenticating and
      upgrading: https://www.docker.com/increase-rate-limit

[terrytangyuan]

You can use podSpecPatch to patch each container’s spec. An example here: https://github.com/argoproj/argo-workflows/blob/master/examples/pod-spec-patch.yaml

[saggar]

Thanks for the reply, @terrytangyuan. Seems like KubeFlow doesn't support podSpecPatch. KF only has API support for setting resources.

Do you know of any other way to specify wait container image for KF pipelines?

[terrytangyuan]

That would be a good question/issue to open in KFP repo.