502 error using HTTP2 on GKE ingress

[hadim]

Using GKE 1.17.9-gke.6300.

Consider the following Deployment, Service and Ingress:

# The below spec is an attempt to make argo works over HTTP/2 over the GKE ingress.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: argo-h2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: argo-h2
  template:
    metadata:
      labels:
        app: argo-h2
    spec:
      containers:
        - args:
            [
              server,
              --auth-mode,
              client,
              --namespaced,
              --secure,
              --insecure-skip-verify,
            ]
          image: argoproj/argocli:v2.11.5
          name: argo-h2
          ports:
            - containerPort: 2746
          readinessProbe:
            httpGet:
              path: /
              port: 2746
              scheme: HTTPS
            initialDelaySeconds: 10
            periodSeconds: 20
      # containers:
      #   - name: argo-h2
      #     image: k8s.gcr.io/echoserver:1.10
      #     ports:
      #       - containerPort: 8443
      nodeSelector:
        kubernetes.io/os: linux
      serviceAccountName: argo-remote
---
apiVersion: v1
kind: Service
metadata:
  name: argo-h2
  annotations:
    cloud.google.com/app-protocols: '{"h2-port":"HTTP2"}'
  labels:
    app: argo-h2
spec:
  type: NodePort
  ports:
    - name: h2-port
      port: 2746
      targetPort: 2746
      protocol: TCP
  selector:
    app: argo-h2
---
apiVersion: networking.gke.io/v1beta2
kind: ManagedCertificate
metadata:
  name: argo-remote-certificate-dev
spec:
  domains:
    - xxx.xxxx.com
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: argo-ingress-h2
  annotations:
    kubernetes.io/ingress.global-static-ip-name: xxxx-dev-ip
    networking.gke.io/managed-certificates: argo-remote-certificate-dev
spec:
  backend:
    serviceName: argo-h2
    servicePort: 2746

Accessing the service using a kubectl proxy the argo-h2 service works well. But trying to access from the ingress endpoint using the xxxx-dev-ip IP or xxx.xxxx.com URL returns a 502 error.

If I replace argoproj/argocli:v2.11.5 by k8s.gcr.io/echoserver:1.10, the ingress endpoint works as excepted (as mentioned in the GKE HTTP2 doc).

Any idea how to debug that? It's hard to find relevant logs and understand where in the networking pipeline the issue is.

xref with argoproj/argo-workflows#4297

[hadim]

In fact the server I am using does not support HTTP2 xD