arogcd cluster add command reports error

[par97]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

I have 3 minikube clusters running on a Fedora release 39 machine:

$ minikube profile list
|---------|-----------|------------|-----------------|------|---------|---------|-------|--------|
| Profile | VM Driver |  Runtime   |       IP        | Port | Version | Status  | Nodes | Active |
|---------|-----------|------------|-----------------|------|---------|---------|-------|--------|
| rdr-dr1 | kvm2      | containerd | 192.168.122.164 | 8443 | v1.28.3 | Running |     1 |        |
| rdr-dr2 | kvm2      | containerd | 192.168.122.51  | 8443 | v1.28.3 | Running |     1 |        |
| rdr-hub | kvm2      | containerd | 192.168.122.104 | 8443 | v1.28.3 | Running |     1 |        |
|---------|-----------|------------|-----------------|------|---------|---------|-------|--------|

after install argocd into rdr-hub cluster argocd namespace, I login to it and try to add cluster.

$ argocd login --core
Context 'kubernetes' updated

$ argocd cluster add rdr-dr1 -y
INFO[0000] ServiceAccount "argocd-manager" created in namespace "kube-system"
INFO[0000] ClusterRole "argocd-manager-role" created
INFO[0000] ClusterRoleBinding "argocd-manager-role-binding" created
INFO[0005] Created bearer token secret for ServiceAccount "argocd-manager"
ERRO[0005] finished unary call with code Unknown         error="NOAUTH Authentication required." grpc.code=Unknown grpc.method=Create grpc.service=cluster.ClusterService grpc.start_time="2024-05-31T17:28:51+08:00" grpc.time_ms=177.03 span.kind=server system=grpc
FATA[0005] rpc error: code = Unknown desc = NOAUTH Authentication required.

but the cluster rdr-dr1 is actually added, but just with no version, status info. see:

$ argocd cluster list
SERVER                          NAME        VERSION  STATUS  MESSAGE  PROJECT
https://192.168.122.164:8443    rdr-dr1
https://kubernetes.default.svc  in-cluster

And no error message could be found in any of the argocd pods.

Later I create application CR on rdr-hub cluster, and argocd could deploy this application to rdr-dr1 cluster without issue.

So why it reports error of "NOAUTH Authentication required“ ？

Version

$ argocd version
argocd: v2.11.2+25f7504
  BuildDate: 2024-05-23T13:58:43Z
  GitCommit: 25f7504ecc198e7d7fdc055fdb83ae50eee5edd0
  GitTreeState: clean
  GoVersion: go1.21.10
  Compiler: gc
  Platform: linux/amd64
argocd-server: v2.11.2+25f7504
  BuildDate: 2024-05-23T13:58:43Z
  GitCommit: 25f7504ecc198e7d7fdc055fdb83ae50eee5edd0
  GitTreeState: clean
  GoVersion: go1.21.10
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v5.3.0 2023-12-07T10:45:14Z
  Helm Version: v3.11
  Kubectl Version: v0.26.11
  Jsonnet Version: v0.20.0

[agaudreault]

What was your kube context at the moment you added the cluster? See

argo-cd/cmd/argocd/commands/cluster.go

Lines 53 to 54 in 2639c59

	# Add a target cluster configuration to ArgoCD. The context must exist in your kubectl config:
	argocd cluster add example-cluster

You might want to join the CNCF slack to ask usage questions in https://cloud-native.slack.com/archives/C01TSERG0KZ instead of bug reports. https://github.com/argoproj/argo-cd/discussions is also a good place if you don't have access to Slack.

[par97]

What was your kube context at the moment you added the cluster? See

I was using the rdr-hub context and in the argocd namespace, that is why I still could run login --core and add another cluster.

Thanks for the slack info, I could also ask in the slack channel.

[agaudreault]

NOAUTH Authentication required seems to be a redis error. There was changes in the latest version of argocd to add basic authentication. A password should have been generated automatically in the argocd-redis secret by the argocd-redis-ha deployment init container. You can check if REDIS_PASSWORD is defined in your argocd-server deployment as a variable. Make sure you use at least version 7.1.1 of https://github.com/argoproj/argo-helm/tree/main/charts/argo-cd, or deployment manifests that will configure redis authentication.

[par97]

In the argocd-server pod definition, I see it has following:

  containers:
  - args:
    - /usr/local/bin/argocd-server
    env:
    - name: REDIS_PASSWORD
      valueFrom:
        secretKeyRef:
          key: auth
          name: argocd-redis

Secret argocd-redis exists in the argocd namespace, contains data/auth key and value.

Argocd cluster add command should atuo read this secret to access redis server, right?

[par97]

btw, argocd is deployed using:
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

I followed document of https://argo-cd.readthedocs.io/en/stable/getting_started/

[agaudreault]

In the steps above, you login to Argo CD in core mode, but it seems like you installed the full manifest. I think if you remove the --core, it should work normally. But you might have found a problem with the CLI usage in core mode.

$ argocd login --core
Context 'kubernetes' updated

[leoluz]

btw, argocd is deployed using: kubectl create namespace argocd kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

I followed document of https://argo-cd.readthedocs.io/en/stable/getting_started/

@agaudreault is correct. If you installed all vended manifests (install.yaml), there is no reason for you to run the Argo CD CLI in core mode (--core). That is only required if you are just deploying the Argo CD controller without the API server.

[par97]

Today I created two new minikube clusters, and installed argocd core in the rdr-hub cluster, by using file of https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/core-install.yaml

I checked, this time in the argocd ns, I only have below pods:

$ kubectl get pod
NAME                                                READY   STATUS    RESTARTS   AGE
argocd-application-controller-0                     1/1     Running   0          19m
argocd-applicationset-controller-7dc76d94b4-8thzw   1/1     Running   0          19m
argocd-redis-545f8544bb-5sxrf                       1/1     Running   0          19m
argocd-repo-server-7c55f55c7-7vwk7                  1/1     Running   0          19m

After login --core and add the other cluster in, it reports the same issue.

$ argocd login --core
Context 'kubernetes' updated

$ argocd cluster add rdr-dr1
WARNING: This will create a service account `argocd-manager` on the cluster referenced by context `rdr-dr1` with full cluster level privileges. Do you want to continue [y/N]? y
INFO[0003] ServiceAccount "argocd-manager" created in namespace "kube-system"
INFO[0003] ClusterRole "argocd-manager-role" created
INFO[0003] ClusterRoleBinding "argocd-manager-role-binding" created
INFO[0008] Created bearer token secret for ServiceAccount "argocd-manager"
ERRO[0009] finished unary call with code Unknown         error="NOAUTH Authentication required." grpc.code=Unknown grpc.method=Create grpc.service=cluster.ClusterService grpc.start_time="2024-06-06T12:20:35+05:30" grpc.time_ms=206.557 span.kind=server system=grpc
FATA[0009] rpc error: code = Unknown desc = NOAUTH Authentication required.

$ argocd cluster list
SERVER                          NAME        VERSION  STATUS  MESSAGE  PROJECT
https://192.168.22.71:8443      rdr-dr1
https://kubernetes.default.svc  in-cluster

So this does look like a bug, even I only installed with core mode.