Proxy env variables on argocd-server breaks repos connectivity

[u72z]

Environment:
ArgoCD is deployed on a K8s cluster in a corporate environment, behind a standard http proxy :3128.
Username and password are passed for basic auth in the proxy variables HTTP_PROXY and HTTPS_PROXY (both use http).
NO_PROXY is adjusted accordingly.

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

• When the proxy environment variables are set on argocd-server (to allow OIDC with external IdP like Okta via the proxy) the repos won't connect (regardless of the proxy being specified in each repo's config or set globally as environment variable for the pod repo-server):

$ argocd repo add https://github.com/argoproj/argocd-example-apps --proxy $http_proxy
FATA[0000] rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: http2: frame too large" 
$ argocd repo add https://github.com/argoproj/argocd-example-apps
FATA[0000] rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: http2: frame too large"
$ argocd repo add http://github.com/argoproj/argocd-example-apps --proxy $http_proxy --grpc-web
FATA[0000] rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: http2: frame too large" 

• When those proxy env variables are unset on argocd-server, the external repos connect (both if the proxy is set specifically on each repo or if it's configured as environment variable for the pod repo-server). But in that case the OIDC auth with the external provider doesn't (timeout because the argocd-server can't join the IdP)

So basically I have to choose between Okta OIDC Auth or external git/helm repositories in a deployment in this environment.

To Reproduce

Configure the proxy env variables on the argocd-server.

Expected behavior

I expect to have a way to have both connections (OIDC and external repos) working via the proxy in the same deployment

Screenshots

Version

argocd: v2.7.6+00c914a
  BuildDate: 2023-06-20T21:18:20Z
  GitCommit: 00c914a948d9e8ad99be8bd82a368fbdeba12f88
  GitTreeState: clean
  GoVersion: go1.19.10
  Compiler: gc
  Platform: linux/amd64
argocd-server: v2.7.6+00c914a.dirty
  BuildDate: 2023-06-20T20:51:13Z
  GitCommit: 00c914a948d9e8ad99be8bd82a368fbdeba12f88
  GitTreeState: dirty
  GoVersion: go1.19.10
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v5.0.1 2023-03-14T01:32:48Z
  Helm Version: v3.11.2+g912ebc1
  Kubectl Version: v0.24.2
  Jsonnet Version: v0.19.1

Logs

time="2023-06-30T07:09:30Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GetUserInfo grpc.service=session.SessionService grpc.start_time="2023-06-30T07:09:30Z" grpc.time_ms=3.631 span.kind=server system=grpc
time="2023-06-30T07:09:30Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=List grpc.service=project.ProjectService grpc.start_time="2023-06-30T07:09:30Z" grpc.time_ms=8.975 span.kind=server system=grpc
time="2023-06-30T07:09:36Z" level=warning msg="finished unary call with code Unavailable" error="rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: http2: frame too large"" grpc.code=Unavailable grpc.method=Create grpc.service=repository.RepositoryService grpc.start_time="2023-06-30T07:09:36Z" grpc.time_ms=69.633 span.kind=server system=grpc
time="2023-06-30T07:09:37Z" level=warning msg="finished unary call with code Unavailable" error="rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: http2: frame too large"" grpc.code=Unavailable grpc.method=Create grpc.service=repository.RepositoryService grpc.start_time="2023-06-30T07:09:37Z" grpc.time_ms=63.492 span.kind=server system=grpc
time="2023-06-30T07:09:37Z" level=warning msg="finished unary call with code Unavailable" error="rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: http2: frame too large"" grpc.code=Unavailable grpc.method=Create grpc.service=repository.RepositoryService grpc.start_time="2023-06-30T07:09:37Z" grpc.time_ms=59.394 span.kind=server system=grpc
time="2023-06-30T07:09:37Z" level=warning msg="finished unary call with code Unavailable" error="rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: http2: frame too large"" grpc.code=Unavailable grpc.method=Create grpc.service=repository.RepositoryService grpc.start_time="2023-06-30T07:09:36Z" grpc.time_ms=530.524 span.kind=server system=grpc
time="2023-06-30T07:11:16Z" level=warning msg="finished unary call with code Unavailable" error="rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: http2: frame too large"" grpc.code=Unavailable grpc.method=Create grpc.service=repository.RepositoryService grpc.start_time="2023-06-30T07:11:15Z" grpc.time_ms=59.621 span.kind=server system=grpc

Related issues for reference

#2954
#9478
#4333
#2243

[fr3man1]

+1
Having same issue. Thinking, that maybe I will setup idp through dex so only that pod would have HTTP_PROXY