Argo CD Istio deploy compatibility

[senare]

Summary

We are trying to make use of Argo CD and are deploying it together with Istio (service mesh).

However there are a few rough edges with using the provided manifest and deploying it without modifications, so I wonder if they couldn't be address 'upstreams' ?!

This is spec here => https://istio.io/docs/ops/deployment/requirements/

Also Included here for easy ref

Named service ports: Service ports must be named. The port name key/value pairs must have the following syntax: name: [-]. See Protocol Selection for more details.

Deployments with app and version labels: We recommend adding an explicit app label and version label to deployments. Add the labels to the deployment specification of pods deployed using the Kubernetes Deployment. The app and version labels add contextual information to the metrics and telemetry Istio collects.

The app label: Each deployment specification should have a distinct app label with a meaningful value. The app label is used to add contextual information in distributed tracing.

The version label: This label indicates the version of the application corresponding to the particular deployment.
Istio have some required conventions about the port naming i.e need to be on the form -suffix.

I would appreciate some feedback on how to handle this !

(Currently I work around it with copy&paste the manifest and do some minor edits, but its a bit hacky)

[alexec]

Would you like to raise a PR on the manifests to add these?

[senare]

I will make an attempt :)

But might not be the fastest, I need to figure out how to build and test etc ...

[martivo]

In my case enabling istio on argocd broke argo. If someone else ends up on this ticket because argocd won't work with istio. The reason is the same requirements - specifically service port naming. For example when using "--insecure" with argo-cd then you can not use "https" as the port name of argocd-server service, since istio will assume it is https. As a quick workaround I prefixed all the service port names in argocd with "tcp-" and now it is working.

The related errors in argocd were "tls: first record does not look like a TLS handshake".
I am not reporting it as a bug, since when this feature is done then the problem will go away by itself.

[vigohe]

My workaround was to enable "--insecure" at argocd-server and disable "TLS"

VirtualService.yaml

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: argocd-server-vts
  namespace: argocd
spec:
  gateways:
  - default/global-gateway
  hosts:
  - argocd.example.com
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: argocd-server.argocd.svc.cluster.local
        port:
          number: 80

DestinationRule.yaml

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: argocd-server-dtrl
  namespace: argocd
spec:
  host: argocd-server.argocd.svc.cluster.local
  trafficPolicy:
    tls:
      mode: DISABLE

[nemo83]

I wrote about how to bootstrap a k8s cluster with ArgoCD 1.4.2 and Istio 1.5.1 using istio Operator in this blog post: https://nemo83.dev/posts/argocd-istio-operator-bootstrap/
Inside you'll find also link to github repo containing all the charts mentioned in the article as well the config updates to get ArgoCD served via Istio ingress gateway, the only thing missing is https, but if you know istio you should be able to add this missing bit yourself. I will blog about this later.

Please pay attention to:

1. this kustomization https://github.com/speedwing/eks-argocd-bootstrap/tree/master/argocd-bootstrap/argocd-istio-bootstrap
2. This chart where Gateway and Virtual services are configured: https://github.com/speedwing/eks-argocd-bootstrap/tree/master/applications/argocd-istio-app

Hope it helps

[L-U-C-K-Y]

Thanks for the workarounds!

I tried using SSL Passthrough of istio, but without success.
I'm not sure if I could pass-through multiple protocols over istio (gRPC/HTTPS) on the same port.

In my case enabling istio on argocd broke argo. If someone else ends up on this ticket because argocd won't work with istio. The reason is the same requirements - specifically service port naming. For example when using "--insecure" with argo-cd then you can not use "https" as the port name of argocd-server service, since istio will assume it is https. As a quick workaround I prefixed all the service port names in argocd with "tcp-" and now it is working.

The related errors in argocd were "tls: first record does not look like a TLS handshake".
I am not reporting it as a bug, since when this feature is done then the problem will go away by itself.

I would highly appreciate if this could get fixed by the argo team.

[Cajga]

Hi Guys,

I've created a kustomize setup to make the ArgoCD manifests fully compatible with Istio requirements (link in the description of the ticket). With this these are the steps to install ArgoCD on an Istio enabled k8s cluster with Istio TLS passthrough:

• wget -O argocd_install.yaml https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
• create the three files below (kustomizations.yaml, istio_patches.yaml and argocd-gateway.yaml)
• adjust the deployment versions in the istio_patches.yaml based on the image versions in the argocd_install.yaml
• kubectl create namespace argocd
• kubectl label namespace argocd istio-injection=enabled
• kubectl apply -k argocd_install.yaml
• lubectl apply -f argocd-gateway.yaml
• create a hosts file entry with argocd-dev.example.com and your external IP

The kustomize does the following changes:

• enables the following options on arg-server --repo-server argocd-repo-server:8081 --insecure
• adds app and version labels for Deployments
• fixes port names on Services (basically introduces http- prefix where it is needed)
• deletes port 443 from argocd-server service (we will use mTLS from istio anyway)
• puts every resource into the argocd namespace

kustomizations.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: argocd
resources:
  - argocd_install.yaml
patchesStrategicMerge:
  - istio_patches.yaml

istio_patches.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/component: server
    app.kubernetes.io/name: argocd-server
    app.kubernetes.io/part-of: argocd
    app: argocd-server
    version: v1.6.1
  name: argocd-server
spec:
  template:
    metadata:
      labels:
        app: argocd-server
        version: v1.6.1
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/component: repo-server
    app.kubernetes.io/name: argocd-repo-server
    app.kubernetes.io/part-of: argocd
    app: argocd-repo-server
    version: v1.6.1
  name: argocd-repo-server
spec:
  template:
    metadata:
      labels:
        app: argocd-repo-server
        version: v1.6.1
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/component: redis
    app.kubernetes.io/name: argocd-redis
    app.kubernetes.io/part-of: argocd
    app: argocd-redis
    version: v5.0.3
  name: argocd-redis
spec:
  template: 
    metadata:
      labels:
        app: argocd-redis
        version: v5.0.3
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/component: dex-server
    app.kubernetes.io/name: argocd-dex-server
    app.kubernetes.io/part-of: argocd
    app: argocd-dex-server
    version: v2.22.0
  name: argocd-dex-server
spec:
  template:
    metadata:
      labels:
        app: argocd-dex-server
        version: v2.22.0
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/component: application-controller
    app.kubernetes.io/name: argocd-application-controller
    app.kubernetes.io/part-of: argocd
    app: argocd-application-controller
    version: v1.6.1
  name: argocd-application-controller
spec:
  template:
    metadata:
      labels:
        app: argocd-application-controller
        version: v1.6.1
---
apiVersion: v1
kind: Service
metadata:
  name: argocd-dex-server
spec:
  ports:
  - name: http
    port: 5556
    protocol: TCP
    targetPort: 5556
  - name: http-grpc
    port: 5557
    protocol: TCP
    targetPort: 5557
  - name: http-metrics
    port: 5558
    protocol: TCP
    targetPort: 5558

---
apiVersion: v1
kind: Service
metadata:
  name: argocd-metrics
spec:
  ports:
  - name: http-metrics
    port: 8082
    protocol: TCP
    targetPort: 8082
---
apiVersion: v1
kind: Service
metadata:
  name: argocd-repo-server
spec:
  ports:
  - name: https-server
    port: 8081
    protocol: TCP
    targetPort: 8081
  - name: http-metrics
    port: 8084
    protocol: TCP
    targetPort: 8084
---
apiVersion: v1
kind: Service
metadata:
  name: argocd-server-metrics
spec:
  ports:
  - name: http-metrics
    port: 8083
    protocol: TCP
    targetPort: 8083
---
apiVersion: v1
kind: Service
metadata:
  name: argocd-server
spec:
  ports:
  - port: 80
    $patch: delete
  - name: https-argocd-server
    port: 443
    protocol: TCP
    targetPort: 8080

argocd-gateway.yaml

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: argocd-gateway
  namespace: argocd
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    tls:
      httpsRedirect: true
    hosts:
    - "argocd-dev.example.com"
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: PASSTHROUGH
    hosts:
    - "argocd-dev.example.com"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: argocd
  namespace: argocd
spec:
  hosts:
  - "argocd-dev.example.com"
  gateways:
  - argocd-gateway
  tls:
  - match:
    - port: 443
      sniHosts:
      - argocd-dev.example.com
    route:
    - destination:
        host: argocd-server.argocd.svc.cluster.local
        port:
          number: 443

[sp185503]

@Cajga Thanks for posting the fix. Side note, I had to name the port with http-, typical Istio stuff.

---
apiVersion: v1
kind: Service
metadata:
  name: argocd-server
spec:
  ports:
  - name: http-argocd-server
    port: 80
    protocol: TCP
    targetPort: 8080
  - port: 443
    $patch: delete

[Cajga]

@sp185503 well, I do rename it when it was necessary... Did I miss one?
According to the specs it is <protocol>[-<suffix>].
My understanding is that the -<suffix> part is not mandatory (it is not there in some of their examples).

[sp185503]

Honestly I've had so many issues where the name of the port was just http and Istio wouldn't forward traffic correctly. If you search for articles on this you'll find to fix this problem you need to add a suffix...I would be interested to hear other's input on this. Currently I'm running 1.5.x in my environments.

[Cajga]

@sp185503 I just asked if http is enough or we need the -suffix (I also mentioned that you had issues without) on istio slack and got the following answer from @howardjohn (Istio maintainer):

I have never heard of this issue, the code treats them the same (in theory at least)

Maybe you could elaborate a bit what did you see there and we can find out what was the problem.

For me, the kustomize above (after I applied my fixes) works well.

[sp185503]

In my case enabling istio on argocd broke argo. If someone else ends up on this ticket because argocd won't work with istio. The reason is the same requirements - specifically service port naming. For example when using "--insecure" with argo-cd then you can not use "https" as the port name of argocd-server service, since istio will assume it is https. As a quick workaround I prefixed all the service port names in argocd with "tcp-" and now it is working.

The related errors in argocd were "tls: first record does not look like a TLS handshake".
I am not reporting it as a bug, since when this feature is done then the problem will go away by itself.

After following @Cajga kustomize solution I was able to get Argocd installed with Istio however we were getting the following errors when trying to create a new application in Arogocd. Using thie tcp- prefix for all services ended up being the final solution. Here is a sample:

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/component: server
    app.kubernetes.io/name: argocd-server
    app.kubernetes.io/part-of: argocd
  name: argocd-server
  namespace: argocd
spec:
  ports:
  - name: tcp-argocdserver
    port: 80
    protocol: TCP
    targetPort: 8080
  selector:
    app.kubernetes.io/name: argocd-server
  type: ClusterIP

Previous error message:

Unable to create application: application spec is invalid: InvalidSpecError: Unable to get app details: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: authentication handshake failed: tls: first record does not look like a TLS handshake"

[Cajga]

@sp185503 I got that error when the agro-repo-server service had http instead of https in its port name. In the latest kustomize (see above the updated version) I fixed that:

apiVersion: v1
kind: Service
metadata:
  name: argocd-repo-server
spec:
  ports:
  - name: https-server
    port: 8081
    protocol: TCP
    targetPort: 8081
  - name: http-metrics
    port: 8084
    protocol: TCP
    targetPort: 8084

Please note that using tcp prefix will limit the possibilities that istio provides you.
I am working on a PR for get this into argocd manifests. Should be ready in few days.

[sp185503]

@Cajga I'll try that in a few minutes.