GRPC, HTTP 2 does not work

[alexcpn]

What I tried

Environment
Kubernetes cluster v1.7

1. Followed https://haproxy-ingress.github.io/docs/getting-started/ and tested with Nginx Pod - all works
2. Created a test GRPC service

docker tag alexcpn/aa_sample_service_go:1.0 alexcpn/sample-grpc-go:1.0
kubectl create deployment sample-grpc  --image=alexcpn/sample-grpc-go:1.0  -n test
kubectl -n test expose deployment sample-grpc --port=50051

Let's use Kube proxy and test if all is fine from Server and Client side

In server

kubectl port-forward service/sample-grpc --address 0.0.0.0 50051:50051 --namespace test

In Client

use address = "sample-grpc.10.131.232.223.nip.io:50051"
root@docker-desktop:/go/microservice/aa_sample_service_go/test_client# go run client.go
2020/08/18 09:00:31 Greeting 2: Some Valid response from server

All good

3. Now let's create the ingress and test

HOST=sample-grpc.10.131.232.223.nip.io

kubectl create -f - <<EOF
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: sample-grpc
  namespace: test
spec:
  rules:
  - host: $HOST
    http:
      paths:
      - backend:
          serviceName: sample-grpc
          servicePort: 50051
        path: /
EOF

[root@green--1 haproxy]# k -n test get ingress
NAME          HOSTS                               ADDRESS   PORTS   AGE
nginx         nginx.10.131.232.223.nip.io                   80      177m
sample-grpc   sample-grpc.10.131.232.223.nip.io             80      36s

However, the client does not work.

That seems fine; as from https://www.haproxy.com/blog/haproxy-1-9-2-adds-grpc-support/
and from https://haproxy-ingress.github.io/docs/configuration/keys/#backend-protocol we need to add the backend-protocol part

Did that

kubectl -n ingress-controller edit configmap haproxy-ingress

apiVersion: v1
data:
  tls-alpn: h2,http/1.1
  use-htx: "true"
  backend-protocol: h2
kind: ConfigMap
metadata:
  creationTimestamp: "2020-08-18T10:59:12Z"
  name: haproxy-ingress
  namespace: ingress-controller
  resourceVersion: "57398358"
  selfLink: /api/v1/namespaces/ingress-controller/configmaps/haproxy-ingress
  uid: e71c2983-e5b1-4cd2-9d41-6a152a8d51c7
~                  

However GRPC client doe not connect
Expected - GRPC client should connect

What am I am configuring wrong or is it a bug ?

Secondary

Also the nginx is also not working with backend as h2.
There is no option to set both as h2,h1 as from the controller logs this is set as an invalid option.

So is it that if we set backend as h2 , HTTP 1.1 servers won't work?

Note that I am using no certificates in the test, all HTTP and insecure connections. ( nginx controller had this restriction of TLS needed for GRPC, but for HAProxy this is not mentioned anywhere as a restriction.

Some more details

[root@green--1 haproxy]# k -n test describe  ingress sample-grpc
Name:             sample-grpc
Namespace:        test
Address:          
Default backend:  default-http-backend:80 (<none>)
Rules:
  Host                               Path  Backends
  ----                               ----  --------
  sample-grpc.10.131.232.223.nip.io  
                                     /   sample-grpc:50051 (10.244.3.21:50051)
Annotations:
Events:
  Type    Reason  Age   From                Message
  ----    ------  ----  ----                -------
  Normal  CREATE  22m   ingress-controller  Ingress test/sample-grpc
  Normal  UPDATE  22m   ingress-controller  Ingress test/sample-grpc
[root@green--1 haproxy]# kubectl -n ingress-controller get pods
NAME                    READY   STATUS    RESTARTS   AGE
haproxy-ingress-l9cb7   1/1     Running   0          33m

[jcmoraisjr]

Note that I am using no certificates in the test, all HTTP and insecure connections. ( nginx controller had this restriction of TLS needed for GRPC, but for HAProxy this is not mentioned anywhere as a restriction.

Hi, this is a protocol limitation - you need to configure alpn in order to allow server and client agree about the http version they want to speak, there is no way to do that in plain http without breaking http1 clients. Because of that you need https to properly configure h2 in the client side.

You can however do the following trick if you don't have http1 clients: use proto keyword in the bind line, you can do it in a dirty-ugly-what!? way configuring :80 proto h2 in the bind-http global config - haproxy ingress copy this configuration verbatim to the bind in the http section. It should work.

Also the nginx is also not working with backend as h2.

The same problem above - you ask haproxy to speak h2 in plain http, no handshake, no alpn involved, maybe this would work if you either:

• configure nginx to speak h2 as well
• configure tls/secure backend, add the tls stuff in the nginx and configure it to allow speak h2

[alexcpn]

Thanks for the clarification.

configure tls/secure backend, add the tls stuff in the nginx and configure it to allow speak h2

Does it mean that we cannot terminate the TLS at Ingress Controller. But each backend service should handle TLS

[alexcpn]

We want ssl to be terminated at the ingress controller. If we use https at ningx side then we need to configure certificates in niginx (and lot of other services)

So I added http2 to nginx listen at port 80 without configuring certificates at nginx layer

(In nginx pod changed  /etc/nginx/conf.d/default.conf  and did nginx -t and nignx -s reload for now)
server {
    listen       80 http2;
    listen  [::]:80 http2;

In HA Proxy Ingress controller I added a certificate and keep this configuration - backend as http2 (h2)

apiVersion: v1
data:
  backend-protocol: h2
  ssl-certificate: secret/test-cert
  tls-alpn: h2,http/1.1

Now I am able to connect to nginx with HTTP and HTTPS

$ curl -ksI  --h
ttp2 https://nginx.10.131.232.223.nip.io/
HTTP/2 200
server: nginx/1.19.2
date: Thu, 20 Aug 2020 09:17:20 GMT
content-type: text/html
content-length: 612
last-modified: Tue, 11 Aug 2020 15:16:45 GMT
etag: "5f32b65d-264"
accept-ranges: bytes
strict-transport-security: max-age=15768000

$ curl -ksI  --http2 http://nginx.10.131.232.223.nip.io/
HTTP/1.1 200
server: nginx/1.19.2
date: Thu, 20 Aug 2020 09:17:41 GMT
content-type: text/html
content-length: 612
last-modified: Tue, 11 Aug 2020 15:16:45 GMT
etag: "5f32b65d-264"
accept-ranges: bytes
strict-transport-security: max-age=15768000

So the flow is like this

HTTPS (HTTP2) --> TLS Terminated at Ingres Controller --> HTTP2 -- >NGINX
HTTP (HTTP2) --> Ingres Controller --> HTTP2 -- >NGINX

However, with the GRPC Server this is not working

GRPC HTTPS (HTTP2) --> TLS Terminated at Ingres Controller --> HTTP2 -- >sample_server

Client
conn, err :=  grpc.Dial(address, grpc.WithTransportCredentials(credentials.NewClientTLSFromCert(nil, "")))

Error in client on connection

"transport: authentication handshake failed: tls: first record does not look like a TLS handshake"

[jcmoraisjr]

Does it mean that we cannot terminate the TLS at Ingress Controller. But each backend service should handle TLS

No, you can terminate and reencrypt the connection.

If we use https at ningx side then we need to configure certificates in niginx (and lot of other services)

Yup, this is a side effect of having both h1 and h2 working in the backend side. You can however use distinct paths or distinct domains to h1 and h2 services and leave this crt/key stuff only in the ingress side.

"transport: authentication handshake failed: tls: first record does not look like a TLS handshake"

Double check if you are in fact connecting in the https/tls port, defaults to :443. What's the output of a curl -kv https://<ingress>:<tls-port>?

[alexcpn]

I changed to :443 and I am now getting a connection thanks for the support

Client 
address = "sample-grpc.10.131.232.223.nip.io:443"

config := &tls.Config{
		InsecureSkipVerify: true, --> true means do not check 
		// else you will get an error -- ransport: authentication handshake failed: x509: certificate is valid for ingress.local, not sample-grpc.10.131.232.223.nip.io"
}

Output

root@docker-desktop:/go/microservice/aa_sample_service_go/test_client# go run client.go
2020/08/20 11:53:23 Greeting 2: Some Valid response from server

curl -kv  --ht
tp2 https://sample-grpc.10.131.232.223.nip.io:443/
*   Trying 10.131.232.223:443...
* TCP_NODELAY set
* Connected to sample-grpc.10.131.232.223.nip.io (10.131.232.223) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
*  start date: Aug 18 10:59:17 2020 GMT
*  expire date: Aug 18 10:59:17 2021 GMT
*  issuer: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
> GET / HTTP/1.1
> Host: sample-grpc.10.131.232.223.nip.io
> User-Agent: curl/7.68.0
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.3 (IN), TLS alert, close notify (256):
* Empty reply from server
* Closing connection 0
* TLSv1.3 (OUT), TLS alert, close notify (256):
curl: (52) Empty reply from server

[alexcpn]

Also the nginx is also not working with backend as h2.
The same problem above - you ask haproxy to speak h2 in plain http, no handshake, no alpn involved, maybe this would work if you either:

configure nginx to speak h2 as well
configure tls/secure backend, add the tls stuff in the nginx and configure it to allow speak h2

I could make nginx speak h2 and it worked. Would this work if we use HTTPS in ingress controller and TLS terminated at ingress controller, and nginx only speaks HTTP1.1

[jcmoraisjr]

The only way to mix h1 and h2 connections in the same configuration is using tls and alpn. If you use plain http you need to choose one or another, and both client and server need to choose the same protocol version. Frontend and backend configurations are also unrelated, so what you configure in the frontend does not reflect in what the backend will do and vice versa.

[alexcpn]

The only way to mix h1 and h2 connections in the same configuration is using tls and alpn

I used the below configuration (with tls and alpn) and tested with two nginx services, one configured to serve HTTP2 and the other default - HTTP1.1.
With the below configuration only the nginx server which is configured to server HTTP2 worked. The other fails

So is it that , currently it is impossible for haproxy to handle both HTTP1.1 and HTTP2 in the backend

 apiVersion: v1
data:
  backend-protocol: h2
  ssl-certificate: secret/test-cert
  tls-alpn: h2,http/1.1
  use-htx: "true

Note I also tried putting h2-ssl. But only HTTP1.1 servers worked in this case. When is this option used ?

more detailed output here - https://github.com/alexcpn/alexcpn.github.io/blob/master/html/other/haproxy-grpc.md

[jcmoraisjr]

Note that tls-alpn does the alpn configuration in the frontend - so you can use both h1 and h2 requests to haproxy. The backend side is another story and you globally configured to use h2. Currently haproxy ingress doesn't have a syntax sugar to configure alpn per backend but you can use config-backend and configure a default-server yourself with haproxy's alpn option. Note also that you need to configure secure-backend otherwise the alpn configuration will not take effect.