ArgoCD can login via UI, but cannot login via CLI

[esn89]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

I can login through the UI but cannot login via the CLI with argocd.

My set up is a bit special:

1. I am not exposing it via a public endpoint
2. I am using an Internal Loadbalancer on GKE with a private routable IP. This works as I can access and login via the UI on the browser.
3. I have a self signed TLS cert (for internal usage from my company) which the ILB uses.

This is a bit of a blocker because I wish to use argocd cluster add so I can add my other clusters.

When I login it does this:

argocd login argocd.myurl.local --grpc-web-root-path=/argocd --username=admin --password=123abc
WARNING: server certificate had error: x509: certificate signed by unknown authority. Proceed insecurely (y/n)? y
FATA[0001] rpc error: code = Unknown desc = POST https://argocd.myurl.local:443/argocd/session.SessionService/Create failed with status code 404 

The :443 added at the end looks a bit suspicious.

To Reproduce

Install ArgoCD v2.7.4+a33baa3.dirty

Kubernetes service:

apiVersion: v1
kind: Service
metadata:
  annotations:
    cloud.google.com/backend-config: '{"ports": {"http":"argocd-server"}}'
    cloud.google.com/load-balancer-type: internal
    cloud.google.com/neg: '{"ingress": true}'
    cloud.google.com/neg-status: '{"network_endpoint_groups":{"80":"k8s1-aaa123123"},"zones":["us-central1-a","us-central1-b","us-central1-f"]}'
    meta.helm.sh/release-name: argocd
    meta.helm.sh/release-namespace: argocd
  labels:
    app.kubernetes.io/component: server
    app.kubernetes.io/instance: argocd
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: argocd-server
    app.kubernetes.io/part-of: argocd
    helm.sh/chart: argo-cd-5.35.1
  name: argocd-server
  namespace: argocd
spec:
  clusterIP: 172.21.53.58
  clusterIPs:
  - 172.21.53.58
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 8080
  - name: https
    port: 443
    protocol: TCP
    targetPort: 8080
  selector:
    app.kubernetes.io/instance: argocd
    app.kubernetes.io/name: argocd-server
  sessionAffinity: None
  type: ClusterIP

Kubernetes Frontend Config:

apiVersion: networking.gke.io/v1beta1
kind: FrontendConfig
metadata:
 annotations:
   meta.helm.sh/release-name: argocd
   meta.helm.sh/release-namespace: argocd
 generation: 1
 labels:
   app.kubernetes.io/component: server
   app.kubernetes.io/instance: argocd
   app.kubernetes.io/managed-by: Helm
   app.kubernetes.io/name: argocd-server
   app.kubernetes.io/part-of: argocd
   helm.sh/chart: argo-cd-5.35.1
 name: argocd-server
 namespace: argocd
spec:
 redirectToHttps:
   enabled: true

Kubernetes BackendConfig:

apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  annotations:
    meta.helm.sh/release-name: argocd
    meta.helm.sh/release-namespace: argocd
  generation: 1
  labels:
    app.kubernetes.io/component: server
    app.kubernetes.io/instance: argocd
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: argocd-server
    app.kubernetes.io/part-of: argocd
    helm.sh/chart: argo-cd-5.35.1
  name: argocd-server
  namespace: argocd
spec:
  healthCheck:
    checkIntervalSec: 30
    healthyThreshold: 1
    port: 8080
    requestPath: /healthz
    timeoutSec: 5
    type: HTTP
    unhealthyThreshold: 2

Kubernetes Ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/backends: '{"k8s1-aaa123123":"HEALTHY","k8s1-aaa123123123212321":"HEALTHY"}'
    ingress.kubernetes.io/https-forwarding-rule: k8s2-fs-argocd
    ingress.kubernetes.io/https-target-proxy: k8s2-ts-argocd
    ingress.kubernetes.io/ssl-cert: k8s2-cr-argocd
    ingress.kubernetes.io/url-map: k8s2-um-argocd
    kubernetes.io/ingress.allow-http: "false"
    kubernetes.io/ingress.class: gce-internal
    kubernetes.io/ingress.regional-static-ip-name: argocd
    meta.helm.sh/release-name: argocd
    meta.helm.sh/release-namespace: argocd
  finalizers:
  - networking.gke.io/ingress-finalizer-V2
  generation: 1
  labels:
    app.kubernetes.io/component: server
    app.kubernetes.io/instance: argocd
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: argocd-server
    app.kubernetes.io/part-of: argocd
    helm.sh/chart: argo-cd-5.35.1
  name: argocd-server
  namespace: argocd
spec:
  rules:
  - host: argocd.myurl.local
    http:
      paths:
      - backend:
          service:
            name: argocd-server
            port:
              number: 80
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - argocd.myurl.local
    secretName: argocd-tls

Kubernetes configmap:

apiVersion: v1
data:
  applicationsetcontroller.enable.leader.election: "true"
  applicationsetcontroller.enable.progressive.syncs: "false"
  applicationsetcontroller.log.format: text
  applicationsetcontroller.log.level: info
  applicationsetcontroller.policy: sync
  controller.log.format: text
  controller.log.level: info
  controller.operation.processors: "10"
  controller.repo.server.timeout.seconds: "60"
  controller.self.heal.timeout.seconds: "5"
  controller.status.processors: "20"
  otlp.address: ""
  repo.server: argocd-repo-server:8081
  reposerver.log.format: text
  reposerver.log.level: info
  reposerver.parallelism.limit: "0"
  server.basehref: /argocd
  server.dex.server: https://argocd-dex-server:5556
  server.dex.server.strict.tls: "false"
  server.disable.auth: "false"
  server.enable.gzip: "true"
  server.insecure: "true"
  server.log.format: text
  server.log.level: info
  server.repo.server.strict.tls: "false"
  server.rootpath: /argocd
  server.staticassets: /shared/app
  server.x.frame.options: sameorigin
kind: ConfigMap

Expected behavior
I expect to be able to login like this:
argocd login argocd.myurl.local --grpc-web-root-path=/argocd --username=admin --password=123abc

However, I get this:

WARNING: server certificate had error: x509: certificate signed by unknown authority. Proceed insecurely (y/n)? y
FATA[0001] rpc error: code = Unknown desc = POST https://argocd.myurl.local:443/argocd/session.SessionService/Create failed with status code 404 

Version

argocd: v2.7.4+a33baa3
  BuildDate: 2023-06-05T19:16:50Z
  GitCommit: a33baa301fe61b899dc8bbad9e554efbc77e0991
  GitTreeState: clean
  GoVersion: go1.19.9
  Compiler: gc
  Platform: linux/amd64
FATA[0000] Argo CD server address unspecified   

Logs

Paste any relevant application logs here.

[mnasruul]

argocd login domain.com --username admin --password 'password' --insecure --grpc-web --grpc-web-root-path "argocd"
dont forget to config file deployment command from argocd-server add --insecure and --rootpath "/argocd"

[gerardnico]

Thanks @mnasruul I got the same problem and it worked only with:

argocd login domain.com --username admin --password 'password' --insecure

I have a basic install with a staging cert of letsencrypt.