Not able to expose grpc app though route still showing http1

[aaryabhatt]

Describe the bug
Hi All,

I am trying to expose a grpc app through route but could not get success. I build a new image with latest version of nginx 1.9.x.
I created a new router on other node with default router.
To reproduce
Steps to reproduce the behavior:

1. Build a new image with latest nginx 1.9.x version.

2. deploy the router.
   oc adm router router-nginx --images=docker-registry.default.svc:5000/openshift/nginx-openshift-router:0.3 --type='' --selector='router=nginx' --force-subdomain='${name}-${namespace}.testingrouter.com

3. Run a sample grpc app, which is working fine when I try to access through service name, from one of node of OCP cluster. As shown below.
   [root@app2dev ~]# docker run namely/grpc-cli ls grpc-pod.test-secret.svc.cluster.local:50051
   grpc.health.v1.Health
   grpc.reflection.v1alpha.ServerReflection
   helloworld.Greeter

4. I add this env variable to dc of nginx router, showing on ouputt of oc describe dc/router-nginx
   ROUTER_USE_HTTP2: true

5. This is my route
   [root@bastiondev ~]# oc describe route grpc-pod
   Name: grpc-pod
   Namespace: test-secret
   Created: 18 hours ago
   Labels:
   Annotations: nginx.router.openshift.io/grpc=true
   openshift.io/host.generated=true
   Requested Host: grpc-pod-test-secret.appseaadev.iamdg.net.ma
   exposed on router router 18 hours ago
   grpc-pod-test-secret.testingrouter.com exposed on router router-nginx 18 hours ago
   Path:
   TLS Termination: passthrough
   Insecure Policy:
   Endpoint Port: grpc-port

Service: grpc-pod
Weight: 100 (100%)
Endpoints: 172.251.13.223:50051
[root@bastiondev ~]#

6. Already enabled TLS passthrough
   [root@bastiondev ~]# oc get route
   NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
   grpc-pod grpc-pod-test-secret.appseaadev.iamdg.net.ma ... 1 more grpc-pod grpc-port passthrough None

Getting below error.
[root@app2dev ~]# docker run --add-host grpc-pod-test-secret.testingrouter.com:10.80.243.13 namely/grpc-cli ls grpc-pod-test-secret.testingrouter.com
Received an error when querying services endpoint.
ServerReflectionInfo rpc failed. Error code: 14, message: failed to connect to all addresses, debug info: {"created":"@1611892660.199138688","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":4143,"referenced_errors":[{"created":"@1611892660.199125817","description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":398,"grpc_status":14}]}
[root@app2dev ~]#

10.80.243.13, this is the IP on which nginx-router running. and I am adding entry for dns in /etc/hosts with --add-host in above command.

Expected behavior
It should return output like below.
[root@app2dev ~]# docker run --add-host grpc-pod-test-secret.testingrouter.com:10.80.243.13 namely/grpc-cli ls grpc-pod-test-secret.testingrouter.com
grpc.health.v1.Health
grpc.reflection.v1alpha.ServerReflection
helloworld.Greeter

It is still serving http1, I tried with bloomrpc to access it, return this error.
{
"error": "14 UNAVAILABLE: Trying to connect an http1.x server"
}

Your environment
Dev environment

• NGINX Router version
• Nginx 1.9.x
• Openshift version
• 3.11
• NGINX or NGINX Plus version
  Nginx
  Additional context
  Add any other context about the problem here. Any log files you want to share.

What I am missing, Could you help to fix this.

--
BR

Amit Bondwal