cannot change executor-image argoproj/argoexec:v2.3.0 -- misconfiguration, 2 controllers

[zhangchao]

Hello, team! we try to use argo workflow in private cluster (private docker hub: artifactory.chao.com/argoproj/v331) . when we submit simple workflow, we found that wait container always use argoproj/argoexec:v2.3.0. failure because we cannot connect to public docker hub.

• controller

apiVersion: apps/v1
kind: Deployment
metadata:
  name: workflow-controller
spec:
  selector:
    matchLabels:
      app: workflow-controller
  template:
    metadata:
      labels:
        app: workflow-controller
    spec:
      containers:
      - args:
        - --configmap
        - workflow-controller-configmap
        - --executor-image
        - "artifactory.chao.com/argoproj/v331/argoexec:v3.3.1"

• workflow

Status:         Pending
IP:
Controlled By:  Workflow/chao-demo-mc2q8
Containers:
  wait:
    Container ID:
    Image:         argoproj/argoexec:v2.3.0
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      argoexec
      wait
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:
      ARGO_POD_NAME:                    chao-demo-mc2q8 (v1:metadata.name)
      ARGO_CONTAINER_RUNTIME_EXECUTOR:  k8sapi
    Mounts:
      /argo/podmetadata from podmetadata (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-drrmm (ro)
  main:
    Container ID:
    Image:         artifactory.chao.com/argoproj/v331/python:alpine3.6
    Image ID:
    Port:          <none>

we set configmap . and kubectl apply, it looks has same issue

• configmap

  executor: |
    image: artifactory.chao.com/argoproj/v331/argoexec:v3.3.1
    resources:
      requests:
        cpu: 10m
        memory: 64Mi

• log

time="2022-04-06T11:55:52.621Z" level=info msg="Get configmaps 200"
time="2022-04-06T11:55:52.632Z" level=info msg="Configuration:\nartifactRepository:\n  s3:\n    accessKeySecret:\n      key: accesskey\n      name: my-minio-cred\n    bucket: my-bucket\n    endpoint: minio:9000\n    insecure: true\n    secretKeySecret:\n      key: secretkey\n      name: my-minio-cred\ncontainerRuntimeExecutors:\n- name: emissary\n  selector:\n    matchLabels:\n      workflows.argoproj.io/container-runtime-executor: emissary\n- name: pns\n  selector:\n    matchLabels:\n      workflows.argoproj.io/container-runtime-executor: pns\nexecutor:\n  image: artifactory.chao.com/argoproj/v331/argoexec:v3.3.1\n  imagePullPolicy: Always\n  name: \"\

{\"image\":\"argoproj/argoexec:v2.3.0\",\"imageID\":\"\",\"lastState\":{},\"name\":\"wait\",\"ready\":false,\"restartCount\":0,\"started\":false,\"state\":{\"waiting\":{\"reason\":\"ContainerCreating\"}}}]}}

[alexec]

This is probably misconfiguration of some kind. Are there two installations in your cluster? One v3.3.1 and one v2.3.0?

[zhangchao]

@terrytangyuan I think that it should be override, --executor-image "artifactory.chao.com/argoproj/v331/argoexec:v3.3.1"

@alexec yes , we have two projects, one project install kubeflow , and another one just instal argoworkflow 3.3.1 .. but I think that it should use different configmap

func (wfc *WorkflowController) executorImage() string {
	if wfc.cliExecutorImage != "" {
		return wfc.cliExecutorImage
	}
	if v := wfc.Config.GetExecutor().Image; v != "" {
		return v
	}
	return fmt.Sprintf("quay.io/argoproj/argoexec:" + argo.ImageTag())
}

[alexec]

User should still be able to specify the executor, as per the code block above.

[terrytangyuan]

Oh I misread the code. @alexec is right that you should be able to specify it. The last line only applies if you don't specify the image explicitly.

[agilgur5]

It was confirmed to be this problem exactly below

[zhangchao]

@alexec @terrytangyuan I guess that it is bug , the problems was caused by argoworkflow 3.3.1 and kubeflow's argo 2.3.0
although they are different namespace , but it looks that two controllers are both watch same event and run.. not sure why

[agilgur5]

You probably installed both Controllers at cluster-scope or have them otherwise watching the same namespaces, so indeed that would result in a conflict like that

[rupert648]

Would it be possible to expand the documentation surrounding specifiying custom image locations for the argo server install? I feel the use case is common but the solution is difficult to discover

[venushka]

For setting the image repository, adding this to the configmap Helm overrides worked for me.

executor:
  image:
    registry: quay-proxy.mydomain.foo

This ends up being --executor-image in the controller deployment

    spec:
      containers:
      - args:
        - --executor-image
        - quay-proxy.mydomain.foo/argoproj/argoexec:v3.5.7