gRPC example

[pcj]

Related to #39 & #2207, I'm psyched to try #2223 + #2246. However, I'm hoping we can use this issue to define an example set of yaml files to use this new feature. I've posted my current set of nginx-ingress config files and hoping to get clarification best practices for gRPC use. I've left out the rbac.yaml file as I assume that does not need any changes.

# nginx-ingress-controller.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-ingress-controller
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
        k8s-app: nginx-ingress-lb
  template:
    metadata:
      annotations:
        prometheus.io/scrape: 'true'
        prometheus.io/port: '10254'
      labels:
        k8s-app: nginx-ingress-lb
    spec:
      serviceAccountName: nginx-ingress-serviceaccount
      containers:
        - name: nginx-ingress-controller
          image: gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.15
          args:
             - /nginx-ingress-controller
             - --default-backend-service=default/default-http-backend
             - --default-ssl-certificate=$(POD_NAMESPACE)/tls-certificate
          env:
             - name: POD_NAME
               valueFrom:
                 fieldRef:
                   fieldPath: metadata.name
             - name: POD_NAMESPACE
               valueFrom:
                 fieldRef:
                   fieldPath: metadata.namespace
          ports:
          - name: http
            containerPort: 80
          - name: https
            containerPort: 443

# nginx-ingress-controller-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx-ingress
  namespace: default
spec:
  type: LoadBalancer
  ports:
  ports:
  - name: http
    port: 80
    targetPort: http
  - name: https
    port: 443
    targetPort: https
  selector:
    k8s-app: nginx-ingress-lb

# ingress-default-backend.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: default-http-backend
  labels:
    k8s-app: default-http-backend
  namespace: default
spec:
  replicas: 1
  template:
    metadata:
      labels:
        k8s-app: default-http-backend
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: default-http-backend
        # Any image is permissable as long as:
        # 1. It serves a 404 page at /
        # 2. It serves 200 on a /healthz endpoint
        image: gcr.io/google_containers/defaultbackend:1.0
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: 10m
            memory: 20Mi
          requests:
            cpu: 10m
            memory: 20Mi
---
apiVersion: v1
kind: Service
metadata:
  name: default-http-backend
  namespace: default
  labels:
    k8s-app: default-http-backend
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    k8s-app: default-http-backend

I have a gRPC service defined as follows. dam-app is an insecure gRPC server listening on TCP port 5151. For simplicity, I'm looking to expose 5151 externally that routes grpc traffic 5151:5151 to the service / deployment. I'm using kube-cert-manager for dynamic letsencrypt certificates and would like to terminate TLS at the nginx-ingress level and run insecure grpc services internally (perhaps I should terminate TLS in the POD? Not sure I need that).

# grpc-deploy.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: dam-app
  labels:
    k8s-app: dam-app
  namespace: default
spec:
  replicas: 1
  template:
    metadata:
      labels:
        k8s-app: dam-app
    spec:
      containers:
      - name: dam-app
        image: gcr.io/foo-bar-baz/dam-app:dev
        ports:
        - containerPort: 5151

# grpc-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: dam-service
  namespace: default
  labels:
    k8s-app: dam-app
spec:
  ports:
  - port: 5151
    targetPort: 5151
    name: grpc
  selector:
    k8s-app: dam-app

# grpc-ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: "nginx"
    ingress.kubernetes.io/ssl-redirect: "true"
  name: dam-ingress
  namespace: default
spec:
  rules:
  - host: grpc.example.com
    http:
      paths:
      - backend:
          serviceName: dam-service
          servicePort: 5151
  tls:
  - secretName: grpc.example.com
    hosts:
      - grpc.example.com

Based on PR #2223, looks like all I need to do is:

1. Replace gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.15 with quay.io/aledbf/nginx-ingress-controller:0.348.
2. Add the annotation nginx.ingress.kubernetes.io/grpc-backend: "true" to my gprc-ingress.yaml.

Is that it?

Is a fancier default backend needed?

(any suggestions on tuning up these yaml files also welcome)

One more thing, what is going to be the recommended image going forward?

Thanks for all your work @aledbf and others on this feature!

Edit: changed 0.345 to 0.348 as this is the newest image at https://quay.io/repository/aledbf/nginx-ingress-controller?tab=tags

[pcj]

Update: This appears to be probably working. This app is listening for traditional http requests at 8080 and grpc services at 5151.

nginx-ingress is sending 308 permanent redirect http -> https and routing to the correct grpc backend. I'm still working on getting the client to connect correctly however, but I think this is due to my grpc client SSL credential setup. Also, requests to /admin are getting a 502 Bad Gateway but I am still debugging and suspect that's a fault with my backend, but still looking at that.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: "nginx"
    ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/grpc-backend: "true"
  name: dam-ingress
  namespace: default
spec:
  rules:
  - host: grpc.example.com
    http:
      paths:
      - path: /admin
        backend:
          serviceName: dam-service
          servicePort: http
      - path: /
        backend:
          serviceName: dam-service
          servicePort: grpc
  tls:
  - secretName: grpc.example.com
    hosts:
      - grpc.example.com

apiVersion: v1
kind: Service
metadata:
  name: dam-service
  namespace: default
  labels:
    k8s-app: dam-app
spec:
  ports:
  - port: 80
    targetPort: 8080
    name: http
  - port: 80
    targetPort: 5151
    name: grpc
  selector:
    k8s-app: dam-app

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: dam-app
  labels:
    k8s-app: dam-app
  namespace: default
spec:
  replicas: 1
  template:
    metadata:
      labels:
        k8s-app: dam-app
    spec:
      containers:
      - name: dam-app
        image: gcr.io/foo-bar-baz/dam-app:dev
        ports:
        - containerPort: 5151
          name: grpc
        - containerPort: 8080
          name: http

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-ingress-controller
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
        k8s-app: nginx-ingress-lb
  template:
    metadata:
      annotations:
        prometheus.io/scrape: 'true'
        prometheus.io/port: '10254'
      labels:
        k8s-app: nginx-ingress-lb
    spec:
      serviceAccountName: nginx-ingress-serviceaccount
      containers:
        - name: nginx-ingress-controller
          image: quay.io/aledbf/nginx-ingress-controller:0.348
          args:
             - /nginx-ingress-controller
             - --default-backend-service=default/default-http-backend
             - --default-ssl-certificate=$(POD_NAMESPACE)/tls-certificate
          env:
             - name: POD_NAME
               valueFrom:
                 fieldRef:
                   fieldPath: metadata.name
             - name: POD_NAMESPACE
               valueFrom:
                 fieldRef:
                   fieldPath: metadata.namespace
          ports:
          - name: http
            containerPort: 80
          - name: https
            containerPort: 443

[gertcuykens]

I think you need ingress.kubernetes.io/ssl-passthrough: "true" ? So basically let de grpc server do the tls handshaking instead of the ingress controller?

[aledbf]

@gertcuykens no because then nginx is not involved at all

[pcj]

In my example above the svc.yaml, both ports are using 80 as input, which is a mistake. Here's my updated svc.yaml:

apiVersion: v1
kind: Service
metadata:
  name: dam-service
  namespace: default
spec:
  selector:
    k8s-app: dam-app
  ports:
  - port: 8080
    targetPort: 8080
    name: http
  - port: 5151
    targetPort: 5151
    name: grpc

[gertcuykens]

So basically you start a grpc server with the same nginx tls credentials right? And you don't have to do Dial grpc.WithInsecure() on the client because nginx wil take care of internal and external grpc tls correct?

Can you confirm the following is correct please?

client <---- TLS GRPC external netwerk -----> NGINX ingress controller <---- TLS GRPC kube internal netwerk -----> backend

[pcj]

I am using this for the client:

import(
    ...
    "crypto/x509"
    "google.golang.org/grpc/credentials"
)

pool, err := x509.SystemCertPool()
creds := credentials.NewClientTLSFromCert(pool, "")
grpc.Dial(address, grpc.WithTransportCredentials(creds))

However, I have not yet confirmed a successful connection, but I can curl -v https://grpc.example.com/my.Service/MyMethod and see http/2 traffic related logs in the nginx-ingress instance. I'm trying to find a chunk of time to fully work it out, but I'm not there yet for my particular application.

[gertcuykens]

Thx, do you also use server tls credentials on the backend behind the nginx ingress controller?

[pcj]

AFAIK should not need to. If TLS is terminated at the ingress level, as far as the gRPC service is concerned, it thinks it is running in an insecure environment (which is true, but only inside the protected cluster).

If I'm wrong about that somebody pls correct me.

[gertcuykens]

As long as it's in the cluster it self I prefer performance over security. Just want to be sure that dropping tls doesn't mean switching into something non http2. Don't know enough about grpc if it is always in http2 mode no matter using tls or not. For example my golang web servers drop http2 when tls is disabled

[pcj]

Adding @stanley-cheung as he is both grpc and nginx expert to maybe answer that question

[gertcuykens]

PS I suspect this going to be a popular thread can you always update your first post with for example the port change and so on, so others don't need to read the discussion to peace together the grpc example. Thanks. Maybe also strip down duplicate config parts in followup messages. Thanks

[aledbf]

PS I suspect this going to be a popular thread can you always update your first post with for example the port change and so on, so others don't need to read the discussion to peace together the grpc example.

@pcj @gertcuykens would be awesome if any of you are willing to send a PR to add a gRPC example here

[gertcuykens]

Totally agree, also much cleaner to followup and review

[pcj]

@aledbf Yes good idea I'll plan to push an example.

[pcj]

The good news is that this is definitely working for the gRPC service. My go client is using the SSL config above (using the system cert pool) calling grpc.example.com:443, TLS gets terminated, call gets routed to the upstream service (which is not using TLS), and my unary gRPC method is being executed and return grpc status OK. Sweet!

I suspect that configuring the grpc server to use TLS should work in conjunction with nginx.ingress.kubernetes.io/grpc-backend: "true". In that case I'd want to mount the cert secret in the pod and configure the grpc server with that file (shouldn't be too hard, but haven't tried it).

NOTE TO SELF: remember not to prefix with https, for example https://grpc.example.com:443 won't work for a dial address, and you won't get an informative reason why the connection is not available

spec:
  rules:
  - host: grpc.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: dam-service
          servicePort: grpc

10.128.0.4 - [10.128.0.4] - - [04/Apr/2018:04:33:52 +0000] "POST /my.package/MyMethod HTTP/2.0" 200 115 "-" "grpc-go/1.6.0" 143 0.003 [default-dam-service-grpc] 10.8.2.66:5151 166 0.002 200

The bad news is that I can't seem to get it working when the ingress.yaml also defines a vanilla http ServiceSpec as well (which my initial example here does). I.e. the following does not work:

spec:
  rules:
  - host: grpc.example.com
    http:
      paths:
      - path: /admin
        backend:
          serviceName: dam-service
          servicePort: http
      - path: /my.package.MyService
        backend:
          serviceName: dam-service
          servicePort: grpc

In this case, the call does seem to get routed to the correct backend service (the non-grpc one at 8080), but the nginx-ingress controller seems to be applying http/2. So, I see this:

 [error] 1227#1227: *142479 upstream sent too large http2 frame: 4740180 while reading response header from upstream, client: 10.128.0.4, server: grpc.example.com, request: "GET /admin/index.html HTTP/2.0", upstream: "grpc://10.8.2.67:8080", host: "grpc.example.com"

Therefore, it appears that although the service selection is correct, grpc / http2 is not. @aledbf is is possible to make the grpc_pass nginx directive more selective?