v1.2.0-rc2 does not retrieve http(s) based git repository behind the proxy

[mitsutaka]

Describe the bug

My kubernetes cluster is behind the proxy. argocd-repo-server needs to set HTTP_PROXY, HTTPS_PROXY and NO_PROXY. After upgrading argocd image to v1.2.0-rc2 and apply manifests. All argocd apps' condition returns ComparisonError. an error log is posted at the bottom.

To Reproduce

1. Deploy argocd-repo-server like this manifest which includes proxy environment variables:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/component: repo-server
    app.kubernetes.io/name: argocd-repo-server
    app.kubernetes.io/part-of: argocd
  name: argocd-repo-server
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: argocd-repo-server
  template:
    metadata:
      labels:
        app.kubernetes.io/name: argocd-repo-server
    spec:
      automountServiceAccountToken: false
      containers:
        - command:
            - argocd-repo-server
            - --redis
            - argocd-redis:6379
          image: quay.io/cybozu/argocd:1.2.0-rc2.1
          livenessProbe:
            initialDelaySeconds: 5
            periodSeconds: 10
            tcpSocket:
              port: 8081
          name: argocd-repo-server
          ports:
            - containerPort: 8081
            - containerPort: 8084
          readinessProbe:
            initialDelaySeconds: 5
            periodSeconds: 10
            tcpSocket:
              port: 8081
          env:
            - name: NO_PROXY
              value: argocd-repo-server,argocd-application-controller,argocd-metrics,argocd-server,argocd-server-metrics,argocd-redis,10.0.0.0/8
            - name: HTTP_PROXY
              value: http://squid.internet-egress.svc.cluster.local:3128
            - name: HTTPS_PROXY
              value: http://squid.internet-egress.svc.cluster.local:3128
          volumeMounts:
            - name: tmp
              mountPath: /tmp
            - name: ssh-known-hosts
              mountPath: /app/config/ssh
            - name: tls-certs
              mountPath: /app/config/tls
      volumes:
        - name: tmp
          emptyDir: {}
        - name: ssh-known-hosts
          configMap:
            name: argocd-ssh-known-hosts-cm
        - name: tls-certs
          configMap:
            name: argocd-tls-certs-cm

2. Deploy Application resources.
3. argocd app sync APP.
4. See error.

I already make sure curl https://github.com/cybozu-go/neco-apps.git/info/refs?service=git-upload-pack with proxy works in the argocd-repo-server pod.

Example

It affects any apps.

Expected behavior

argocd-repo-server should retrieve http(s) based git repository behind the proxy.

Screenshots

If applicable, add screenshots to help explain your problem.

Version

argocd: v1.2.0-rc2+5571cf1
  BuildDate: 2019-08-21T15:47:33Z
  GitCommit: 5571cf133323f1c864472480c56e03e9ff3da0b3
  GitTreeState: clean
  GoVersion: go1.12.6
  Compiler: gc
  Platform: linux/amd64
argocd-server: v1.2.0-rc2+5571cf1
  BuildDate: 2019-08-21T15:47:33Z
  GitCommit: 5571cf133323f1c864472480c56e03e9ff3da0b3
  GitTreeState: clean
  GoVersion: go1.12.6
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: 0.13.1

Logs

argocd-repo-server

time="2019-09-04T02:10:44Z" level=error msg="finished unary call with code Unknown" error="Get https://github.com/cybozu-go/neco-apps.git/info/refs?service=git-upload-pack: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" grpc.code=Unknown grpc.method=GenerateManifest grpc.request.deadline="2019-09-04T02:11:29Z" grpc.service=repository.RepoServerService grpc.start_time="2019-09-04T02:10:29Z" grpc.time_ms=15000.914 span.kind=server system=grpc

Have you thought about contributing a fix yourself?

I am going to make a PR. I found GetRepoHTTPClient() in the commit 5953080#diff-a78af8f9e3e850815ccda822f9b46922L96-R96 used plain http.Client{} instead of githttp.NewClient(). It seems that drops reading proxy address from the environment variable.

[alexec]

@jannfis FYI

[alexec]

Thank you @mitsutaka, please let us know how you get on and if you need any assistance.

[jannfis]

Oops, how could I miss this use-case. Thanks for the report and the offer for fixing it, @mitsutaka. As @alexec said, let us know if we can assist in any way. That being said, I'll not be able to respond the next 2 weeks most likely tho due to private reasons.

I guess we should also consider HTTP proxy connections for end-to-end testing as well, once this feature is implemented.

[mitsutaka]

@alexec @jannfis Thank you for assistance! I'll make it as soon as possible.

[alexec]

Fixed in v1.3. @alexmt should we backport?

[alexmt]

Sure, let's backport. The 1.2.0 is released already but we can create patch release even if this is the only change. I suggest to wait till Monday in case there are more hot-fixes and then release 1.2.1

[alexec]

Cherry-picked to v1.2.