Argo cli skip-tls-verify does not skip as expected

[sylock]

Summary

When using argo cli targetting argo server exposed by a reverse proxy with a self-signed certificate (edge is the reverse proxy), I get an error saying that the certificate is signed by unknown authority.
I did set the flags --insecure-skip-tls-verify an --insecure-skip-verify

The argo targetted is a development environment. That's why the certificate is not valid.

argo --insecure-skip-verify --insecure-skip-tls-verify -e --kubeconfig /home/<OBFUSCATED>/.kubeconfig --argo-server argo-dev.apps.argo-wf.acc.<OBFUSCATED_DOMAIN>:443 --argo-http1 archive list

FATA[2021-02-03T18:37:00.791Z] Get https://argo-dev.apps.argo-wf.acc.<OBFUSCATED_DOMAIN>:443/api/v1/archived-workflows?listOptions.fieldSelector=metadata.namespace%3Dargo-wf-development: x509: certificate signed by unknown authority

Expected behavior: setting the flags to skip the TLS verifications should skip all and any kind of verification including the authority verification

Diagnostics

What Kubernetes provider are you using?
Openshift 4.5.5 with Kubernetes 1.18.3

What version of Argo Workflows are you running?
argo server: 2.12.7
argo cli: 2.12.7 (see below)

argo: v2.12.7
  BuildDate: 2021-02-01T22:08:27Z
  GitCommit: 5f5150730c644865a5867bf017100732f55811dd
  GitTreeState: clean
  GitTag: v2.12.7
  GoVersion: go1.13
  Compiler: gc
  Platform: linux/amd64

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[alexec]

Please can you follow the HTTP1 instructions here:

https://argoproj.github.io/argo/cli/argo/

[sylock]

I tried another way:

The env variables:

env | egrep -i 'argo|kubeconfig'
ARGO_NAMESPACE=argo-wf-development
ARGO_INSECURE_SKIP_VERIFY=true
ARGO_SECURE=True
KUBECONFIG=/home/<OBFUSCATED>/.kubeconfig
ARGO_SERVER=argo-dev.apps.argo-wf.acc.<OBFUSCATED_DOMAIN>:443

Argo list in verbose mode:

argo list -v
DEBU[2021-02-03T19:14:33.520Z] CLI version                                   version="{v2.12.7 2021-02-01T22:08:27Z 5f5150730c644865a5867bf017100732f55811dd v2.12.7 clean go1.13 gc linux/amd64}"
DEBU[2021-02-03T19:14:33.520Z] Client options                                opts="(argoServerOpts=(url=argo-dev.apps.argo-wf.acc.<OBFUSCATED_DOMAIN>:443,path=,secure=false,insecureSkipVerify=true,http=false),instanceID=)"
I0203 19:14:33.524922   20558 loader.go:375] Config loaded from file:  /home/<OBFUSCATED>/.kubeconfig
DEBU[2021-02-03T19:14:33.525Z]                                               listOpts="&ListOptions{LabelSelector:,FieldSelector:,Watch:false,ResourceVersion:,TimeoutSeconds:nil,Limit:0,Continue:,AllowWatchBookmarks:false,}"
FATA[2021-02-03T19:14:33.530Z] rpc error: code = Unavailable desc = connection closed

[alexec]

Try ARGO_HTTP1=true

[sylock]

I tried 2 other ways with different errors:

With HTTP1 enabled:

These env were set:

export ARGO_SERVER=argo-dev.apps.argo-wf.acc.cloud.smals.be:443
export ARGO_SECURE=true
export ARGO_HTTP1=true
export ARGO_INSECURE_SKIP_VERIFY=true
export ARGO_NAMESPACE=argo-wf-development

Verbose argo list

argo list -v
DEBU[2021-02-03T22:13:10.799Z] CLI version                                   version="{v2.12.7 2021-02-01T22:08:27Z 5f5150730c644865a5867bf017100732f55811dd v2.12.7 clean go1.13 gc linux/amd64}"
DEBU[2021-02-03T22:13:10.799Z] Client options                                opts="(argoServerOpts=(url=argo-dev.apps.argo-wf.acc.cloud.smals.be:443,path=,secure=true,insecureSkipVerify=true,http=true),instanceID=)"
I0203 22:13:10.803961   38147 loader.go:375] Config loaded from file:  /home/<OBFUSCATED>/.kubeconfig
DEBU[2021-02-03T22:13:10.804Z]                                               listOpts="&ListOptions{LabelSelector:,FieldSelector:,Watch:false,ResourceVersion:,TimeoutSeconds:nil,Limit:0,Continue:,AllowWatchBookmarks:false,}"
DEBU[2021-02-03T22:13:10.804Z] curl -X GET -H 'Authorization: ******' -d '' 'https://argo-dev.apps.argo-wf.acc.<OBFUSCATED_DOMAIN>:443/api/v1/workflows/argo-wf-development?'
FATA[2021-02-03T22:13:10.846Z] Get https://argo-dev.apps.argo-wf.acc.<OBFUSCATED_DOMAIN>:443/api/v1/workflows/argo-wf-development?: x509: certificate signed by unknown authority

Without HTTP1:

I thought I should get the same result than my first try (when opening the case) but it seems I don't get the same when using the environment variables in place of the cli flags.

Environment variables:

export ARGO_SERVER=argo-dev.apps.argo-wf.acc.cloud.smals.be:443
export ARGO_SECURE=true
export ARGO_INSECURE_SKIP_VERIFY=true
export ARGO_NAMESPACE=argo-wf-development

Verbose argo list:

argo list
FATA[2021-02-03T22:16:36.865Z] rpc error: code = Unknown desc = OK: HTTP status code 200; transport: received the unexpected content-type "text/html; charset=utf-8"
[smals-nim@bastion-argo-wf ~]$ argo list -v
DEBU[2021-02-03T22:16:45.931Z] CLI version                                   version="{v2.12.7 2021-02-01T22:08:27Z 5f5150730c644865a5867bf017100732f55811dd v2.12.7 clean go1.13 gc linux/amd64}"
DEBU[2021-02-03T22:16:45.932Z] Client options                                opts="(argoServerOpts=(url=argo-dev.apps.argo-wf.acc.<OBFUSCATED_DOMAIN>:443,path=,secure=true,insecureSkipVerify=true,http=false),instanceID=)"
I0203 22:16:45.944055   38508 loader.go:375] Config loaded from file:  /home/<OBFUSCATED>/.kubeconfig
DEBU[2021-02-03T22:16:45.945Z]                                               listOpts="&ListOptions{LabelSelector:,FieldSelector:,Watch:false,ResourceVersion:,TimeoutSeconds:nil,Limit:0,Continue:,AllowWatchBookmarks:false,}"
FATA[2021-02-03T22:16:46.002Z] rpc error: code = Unknown desc = OK: HTTP status code 200; transport: received the unexpected content-type "text/html; charset=utf-8"

[alexec]

This is broken for HTTP1. Need to investigate GRPC.

[alexec]

Odd. GRPC works for me.

[sylock]

I will investigate from my side. My current hypothesis is that my reverse proxy in front of argo cause this grpc problem. And could be also related to #5006.