[CLI] --watch option does not work when using --instanceid & unable to list workflows

[ramanNarasimhan77]

Checklist:

• I've included the version.
• I've included reproduction steps.
• I've included the workflow YAML.
• I've included the logs.

What happened:
I have done the following to setup argo workflow with instanceid:

a) Setup a serviceaccount "workflow", rbac Role and Rolebinding

b) Installed argo using the helm chart from https://github.com/argoproj/argo-helm/tree/master/charts/argo

c) submitted a job by specifying the --instanceid and --watch option

d) cli is indefinitely stuck, does not show the steps/status

e) argo list shows empty results possibly because there is another argo installation in a different namespace and I am not sure if cli is somehow connecting to that. I have exposed argo via ingress, Tried setting server:host as :80 but that did not work

argo list -s <ingress-host>:80
FATA[0001] rpc error: code = Unavailable desc = connection closed

What you expected to happen:
a) argo submit with --instanceid and --watch should show the status of each step in the workflow

b) I should be able to use argo list to list the workflows and then use argo get to see the workflow status

How to reproduce it (as minimally and precisely as possible):

a) Create rbac roles

apiVersion: v1
kind: ServiceAccount
metadata:
  name: workflow
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: workflow-role
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - pods/exec
  verbs:
  - create
  - get
  - list
  - watch
  - update
  - patch
- apiGroups:
  - ""
  resources:
  - configmaps
  verbs:
  - get
  - watch
  - list
- apiGroups:
  - ""
  resources:
  - persistentvolumeclaims
  verbs:
  - create
  - get
  - list
  - delete
- apiGroups:
  - argoproj.io
  resources:
  - workflows
  verbs:
  - get
  - list
  - watch
  - update
  - patch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: workflow-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: workflow-role
subjects:
- kind: ServiceAccount
  name: workflow

b) Install argo using helm chart:
helm install ./argo --name argo-wf --set controller.instanceID.enabled=true --set controller.instanceID.useReleaseName=true --set controller.workflowNamespaces\[0\]=argo-wf --set controller.containerRuntimeExecutor=k8sapi --set singlenamespace=true --set init.serviceAccount=workflow --set minio.install=true --set installCRD=true --set server.ingress.enabled=true --set server.ingress.hosts\[0\]=argo.<clusterdomain> --set minio.ingress.enabled=true --set minio.ingress.hosts\[0\]=argo-minio.<cluster-domain> --namespace argo-wf --debug
c) Submit any workflow with instanceid and serviceaccount, though you can see steps running in the ui, the cli is stuck even when workflow is completed

$ argo submit test.yaml --instanceid argo-wf  --serviceaccount workflow --watch
Name:                seldon-deploy-knjb6
Namespace:           argo-wf
ServiceAccount:      workflow
Status:              Pending
Created:             Sat Jun 13 19:30:52 +0530 (now)
Parameters:
  build-push-image:  true
  deploy-model:      true
  git-revision:      master
  git-repo-url:      https://<mygiturl>.git
  git-secret-name:   git-creds
  docker-repo-url:   <my-docker-repo>

d) In another terminal, I tried to list the workflows

$argo list workflows
NAME   STATUS   AGE   DURATION   PRIORITY

Anything else we need to know?:
I am not sharing the workflow details, as on the UI it is showing that workflow ran successfully
I have attached UI screenshot

Environment:

• Argo version:

$ argo version
argo: v2.8.1
  BuildDate: 2020-05-28T23:40:32Z
  GitCommit: 0fff4b21c21c5ff5adbb5ff62c68e67edd95d6b8
  GitTreeState: clean
  GitTag: v2.8.1
  GoVersion: go1.13.4
  Compiler: gc
  Platform: darwin/amd64

• Kubernetes version :

$ kubectl version -o yaml
clientVersion:
  buildDate: "2020-05-21T14:51:23Z"
  compiler: gc
  gitCommit: 2e7996e3e2712684bc73f0dec0200d64eec7fe40
  gitTreeState: clean
  gitVersion: v1.18.3
  goVersion: go1.14.3
  major: "1"
  minor: "18"
  platform: darwin/amd64
serverVersion:
  buildDate: "2020-01-18T23:24:23Z"
  compiler: gc
  gitCommit: 2e808b7cb054ee242b68e62455323aa783991f03
  gitTreeState: clean
  gitVersion: v1.15.9
  goVersion: go1.12.12
  major: "1"
  minor: "15"
  platform: linux/amd64

• Helm version :

 helm version
Client: &version.Version{SemVer:"v2.16.1", GitCommit:"bbdfe5e7803a12bbdf97e94cd847859890cf4050", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.16.1", GitCommit:"bbdfe5e7803a12bbdf97e94cd847859890cf4050", GitTreeState:"clean"}

• Argo Chart version:

Chart version : argo-0.9.3
App Version: v2.8.0

Other debugging information (if applicable):

On submitting with verbose option I can see that CLI gets stuck at DEBU[0002] Piping events to channel and repeatedly prints level=info msg="Re-establishing workflow watch"

$  argo submit test.yaml --instanceid argo-wf  --serviceaccount workflow --watch -v
DEBU[0000] CLI version                                   version="{v2.8.1 2020-05-28T23:40:32Z 0fff4b21c21c5ff5adbb5ff62c68e67edd95d6b8 v2.8.1 clean go1.13.4 gc darwin/amd64}"
DEBU[0000] Client options                                opts="{{ false false} 0x2175d40 0xc000138280}"
DEBU[0000] Getting the template by name
DEBU[0000] Resolving the template                        base="*v1alpha1.Workflow (namespace=mee-argo-wf,name=)" depth=0 tmpl="*v1alpha1.WorkflowStep (workflow)"
DEBU[0000] Getting the template                          base="*v1alpha1.Workflow (namespace=mee-argo-wf,name=)" depth=0 tmpl="*v1alpha1.WorkflowStep (workflow)"
DEBU[0000] Getting the template by name                  base="*v1alpha1.Workflow (namespace=mee-argo-wf,name=)" depth=0 tmpl="*v1alpha1.WorkflowStep (workflow)"
DEBU[0000] Getting the template by name
DEBU[0000] Resolving the template                        base="*v1alpha1.Workflow (namespace=mee-argo-wf,name=)" depth=0 tmpl="*v1alpha1.WorkflowStep (get-source-code)"
DEBU[0000] Getting the template                          base="*v1alpha1.Workflow (namespace=mee-argo-wf,name=)" depth=0 tmpl="*v1alpha1.WorkflowStep (get-source-code)"
DEBU[0000] Getting the template by name                  base="*v1alpha1.Workflow (namespace=mee-argo-wf,name=)" depth=0 tmpl="*v1alpha1.WorkflowStep (get-source-code)"
DEBU[0000] Getting the template by name                  base="*v1alpha1.Workflow (namespace=mee-argo-wf,name=)" depth=0 tmpl="*v1alpha1.WorkflowStep (get-source-code)"
DEBU[0000] Resolving the template                        base="*v1alpha1.Workflow (namespace=mee-argo-wf,name=)" depth=0 tmpl="*v1alpha1.WorkflowStep (get-source-code)"
DEBU[0000] Getting the template                          base="*v1alpha1.Workflow (namespace=mee-argo-wf,name=)" depth=0 tmpl="*v1alpha1.WorkflowStep (get-source-code)"
DEBU[0000] Getting the template by name                  base="*v1alpha1.Workflow (namespace=mee-argo-wf,name=)" depth=0 tmpl="*v1alpha1.WorkflowStep (get-source-code)"
DEBU[0000] Getting the template by name                  base="*v1alpha1.Workflow (namespace=mee-argo-wf,name=)" depth=0 tmpl="*v1alpha1.WorkflowStep (get-source-code)"
DEBU[0000] Resolving the template                        base="*v1alpha1.Workflow (namespace=mee-argo-wf,name=)" depth=0 tmpl="*v1alpha1.WorkflowStep (get-build-params)"
DEBU[0000] Getting the template                          base="*v1alpha1.Workflow (namespace=mee-argo-wf,name=)" depth=0 tmpl="*v1alpha1.WorkflowStep (get-build-params)"
DEBU[0000] Getting the template by name                  base="*v1alpha1.Workflow (namespace=mee-argo-wf,name=)" depth=0 tmpl="*v1alpha1.WorkflowStep (get-build-params)"
DEBU[0000] Getting the template by name                  base="*v1alpha1.Workflow (namespace=mee-argo-wf,name=)" depth=0 tmpl="*v1alpha1.WorkflowStep (get-build-params)"
DEBU[0000] Resolving the template                        base="*v1alpha1.Workflow (namespace=mee-argo-wf,name=)" depth=0 tmpl="*v1alpha1.WorkflowStep (get-build-params)"
DEBU[0000] Getting the template                          base="*v1alpha1.Workflow (namespace=mee-argo-wf,name=)" depth=0 tmpl="*v1alpha1.WorkflowStep (get-build-params)"
DEBU[0000] Getting the template by name                  base="*v1alpha1.Workflow (namespace=mee-argo-wf,name=)" depth=0 tmpl="*v1alpha1.WorkflowStep (get-build-params)"
DEBU[0000] Getting the template by name                  base="*v1alpha1.Workflow (namespace=mee-argo-wf,name=)" depth=0 tmpl="*v1alpha1.WorkflowStep (get-build-params)"
DEBU[0000] Resolving the template                        base="*v1alpha1.Workflow (namespace=mee-argo-wf,name=)" depth=0 tmpl="*v1alpha1.WorkflowStep (build-and-push)"
DEBU[0000] Getting the template                          base="*v1alpha1.Workflow (namespace=mee-argo-wf,name=)" depth=0 tmpl="*v1alpha1.WorkflowStep (build-and-push)"
DEBU[0000] Getting the template by name                  base="*v1alpha1.Workflow (namespace=mee-argo-wf,name=)" depth=0 tmpl="*v1alpha1.WorkflowStep (build-and-push)"
DEBU[0000] Getting the template by name                  base="*v1alpha1.Workflow (namespace=mee-argo-wf,name=)" depth=0 tmpl="*v1alpha1.WorkflowStep (build-and-push)"
DEBU[0000] Resolving the template                        base="*v1alpha1.Workflow (namespace=mee-argo-wf,name=)" depth=0 tmpl="*v1alpha1.WorkflowStep (build-and-push)"
DEBU[0000] Getting the template                          base="*v1alpha1.Workflow (namespace=mee-argo-wf,name=)" depth=0 tmpl="*v1alpha1.WorkflowStep (build-and-push)"
DEBU[0000] Getting the template by name                  base="*v1alpha1.Workflow (namespace=mee-argo-wf,name=)" depth=0 tmpl="*v1alpha1.WorkflowStep (build-and-push)"
DEBU[0000] Getting the template by name                  base="*v1alpha1.Workflow (namespace=mee-argo-wf,name=)" depth=0 tmpl="*v1alpha1.WorkflowStep (workflow)"
DEBU[0000] Resolving the template                        base="*v1alpha1.Workflow (namespace=mee-argo-wf,name=)" depth=0 tmpl="*v1alpha1.WorkflowStep (workflow)"
DEBU[0000] Getting the template                          base="*v1alpha1.Workflow (namespace=mee-argo-wf,name=)" depth=0 tmpl="*v1alpha1.WorkflowStep (workflow)"
DEBU[0000] Getting the template by name                  base="*v1alpha1.Workflow (namespace=mee-argo-wf,name=)" depth=0 tmpl="*v1alpha1.WorkflowStep (workflow)"
DEBU[0000] Getting the template by name                  base="*v1alpha1.Workflow (namespace=mee-argo-wf,name=)" depth=0 tmpl="*v1alpha1.WorkflowStep (workflow)"
DEBU[0000] Resolving the template                        base="*v1alpha1.Workflow (namespace=mee-argo-wf,name=)" depth=0 tmpl="*v1alpha1.WorkflowStep (get-source-code)"
DEBU[0000] Getting the template                          base="*v1alpha1.Workflow (namespace=mee-argo-wf,name=)" depth=0 tmpl="*v1alpha1.WorkflowStep (get-source-code)"
DEBU[0000] Getting the template by name                  base="*v1alpha1.Workflow (namespace=mee-argo-wf,name=)" depth=0 tmpl="*v1alpha1.WorkflowStep (get-source-code)"
DEBU[0000] Getting the template by name                  base="*v1alpha1.Workflow (namespace=mee-argo-wf,name=)" depth=0 tmpl="*v1alpha1.WorkflowStep (get-source-code)"
DEBU[0000] Resolving the template                        base="*v1alpha1.Workflow (namespace=mee-argo-wf,name=)" depth=0 tmpl="*v1alpha1.WorkflowStep (get-build-params)"
DEBU[0000] Getting the template                          base="*v1alpha1.Workflow (namespace=mee-argo-wf,name=)" depth=0 tmpl="*v1alpha1.WorkflowStep (get-build-params)"
DEBU[0000] Getting the template by name                  base="*v1alpha1.Workflow (namespace=mee-argo-wf,name=)" depth=0 tmpl="*v1alpha1.WorkflowStep (get-build-params)"
DEBU[0000] Getting the template by name                  base="*v1alpha1.Workflow (namespace=mee-argo-wf,name=)" depth=0 tmpl="*v1alpha1.WorkflowStep (get-build-params)"
DEBU[0000] Resolving the template                        base="*v1alpha1.Workflow (namespace=mee-argo-wf,name=)" depth=0 tmpl="*v1alpha1.WorkflowStep (build-and-push)"
DEBU[0000] Getting the template                          base="*v1alpha1.Workflow (namespace=mee-argo-wf,name=)" depth=0 tmpl="*v1alpha1.WorkflowStep (build-and-push)"
DEBU[0000] Getting the template by name                  base="*v1alpha1.Workflow (namespace=mee-argo-wf,name=)" depth=0 tmpl="*v1alpha1.WorkflowStep (build-and-push)"
Name:                seldon-deploy-tsw4j
Namespace:           argo-wf
ServiceAccount:      workflow
Status:              Pending
Created:             Sat Jun 13 19:55:46 +0530 (now)
Parameters:
  build-push-image:  true
  deploy-model:      true
  git-revision:      master
  git-repo-url:      https://<my-git-repo>.git
  git-secret-name:   git-creds
  docker-repo-url:   <my-docker-registry>
DEBU[0001] Client options                                opts="{{ false false} 0x2175d40 0xc0007ce6e0}"
DEBU[0002] Piping events to channel

Argo Server Pod logs

time="2020-06-13T14:22:10Z" level=info msg="Re-establishing workflow watch"
time="2020-06-13T14:23:13Z" level=info msg="Re-establishing workflow watch"
W0613 14:24:04.216428       1 reflector.go:302] pkg/mod/k8s.io/client-go@v0.0.0-20191225075139-73fd2ddc9180/tools/cache/reflector.go:98: watch of *v1.ConfigMap ended with: too old resource version: 307575290 (307578869)
time="2020-06-13T14:24:24Z" level=info msg="Re-establishing workflow watch"
time="2020-06-13T14:25:18Z" level=info msg="Alloc=5951 TotalAlloc=28429 Sys=71616 NumGC=18 Goroutines=60"
time="2020-06-13T14:25:58Z" level=info msg="Re-establishing workflow watch"
time="2020-06-13T14:27:06Z" level=info msg="Re-establishing workflow watch"
time="2020-06-13T14:28:13Z" level=info msg="Re-establishing workflow watch"
time="2020-06-13T14:29:24Z" level=info msg="Re-establishing workflow watch"
time="2020-06-13T14:30:18Z" level=info msg="Alloc=6854 TotalAlloc=34396 Sys=71616 NumGC=20 Goroutines=60"



time="2020-06-13T14:31:09Z" level=info msg="Re-establishing workflow watch"
W0613 14:32:02.222634       1 reflector.go:302] pkg/mod/k8s.io/client-go@v0.0.0-20191225075139-73fd2ddc9180/tools/cache/reflector.go:98: watch of *v1.ConfigMap ended with: too old resource version: 307579909 (307584526)
time="2020-06-13T14:32:53Z" level=info msg="Re-establishing workflow watch"

---

Message from the maintainers:

If you are impacted by this bug please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

[alexec]

Are you testing out Argo? Can I recommend you start with the quick start manifests? (Look in /manifests)

[alexec]

I also note you're using an old version of the UI. Pre v2.8. This is probably a bug in the Helm chart you may wish to raise with that team.

[ramanNarasimhan77]

Are you testing out Argo? Can I recommend you start with the quick start manifests? (Look in /manifests)

Hello,
I have tried out the manifests before. My team is evaluating argoproj components (argo for workflow, argo events for triggering them and argocd for deployments)

I am trying to automate argo helm installation using argocd and was trying to install the helm chart manually first to figure out what values might need to be overridden

[ramanNarasimhan77]

I also note you're using an old version of the UI. Pre v2.8. This is probably a bug in the Helm chart you may wish to raise with that team.

I tried the latest images too, and run into the same issue. I will raise an issue in the argo helm repo.

[ramanNarasimhan77]

@alexec
One followup question, argo list is not working when I use -s with <ingress-host>:<default ingress port 80>

Why does this fail, do I need to enable TLS?

[ramanNarasimhan77]

I am able to reproduce this issue even by using the manifests.

There is an existing argo installation in my cluster, deployed using the helm charts. It's controller is setup to look for workflows in only a specific namespace (controller configmap has the namespace entry)

I have installed argo stable using the manifests provided into another namespace "argo"

kubectl create namespace argo

kubectl apply -f quick-start-postgres.yaml -n argo

I have setup a serviceaccount workflow with necessary permissions.

I edited the workflow controller config map and added the following lines and deleted the controller pod so that the changes take effect

instanceid: my-argo
namespace: argo

Then i tried,
argo submit test.yaml --instanceid my-argo --serviceaccount workflow --namespace argo --watch

Cli output gets stuck with

Name: build-deploy-model-l65qz
Namespace: argo
ServiceAccount: workflow
Status: Pending
Created: Wed Jun 17 11:02:57 +0530 (now)
Parameters:
git-revision: master
git-repo-url: https://.git
git-secret-name: git-creds
docker-repo-url:

[alexec]

#2786

[alexec]

#2786

[alexec]

This has been fixed in v2.9, but this has not been released yet.

[alexec]

You must pass --instanceid to all commands - but that is only supported in v2.9.