-
Notifications
You must be signed in to change notification settings - Fork 8.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
gRPC example #2284
Comments
Update: This appears to be probably working. This app is listening for traditional http requests at 8080 and grpc services at 5151. nginx-ingress is sending 308 permanent redirect http -> https and routing to the correct grpc backend. I'm still working on getting the client to connect correctly however, but I think this is due to my grpc client SSL credential setup. Also, requests to /admin are getting a 502 Bad Gateway but I am still debugging and suspect that's a fault with my backend, but still looking at that. apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: "nginx"
ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/grpc-backend: "true"
name: dam-ingress
namespace: default
spec:
rules:
- host: grpc.example.com
http:
paths:
- path: /admin
backend:
serviceName: dam-service
servicePort: http
- path: /
backend:
serviceName: dam-service
servicePort: grpc
tls:
- secretName: grpc.example.com
hosts:
- grpc.example.com apiVersion: v1
kind: Service
metadata:
name: dam-service
namespace: default
labels:
k8s-app: dam-app
spec:
ports:
- port: 80
targetPort: 8080
name: http
- port: 80
targetPort: 5151
name: grpc
selector:
k8s-app: dam-app apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: dam-app
labels:
k8s-app: dam-app
namespace: default
spec:
replicas: 1
template:
metadata:
labels:
k8s-app: dam-app
spec:
containers:
- name: dam-app
image: gcr.io/foo-bar-baz/dam-app:dev
ports:
- containerPort: 5151
name: grpc
- containerPort: 8080
name: http apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: nginx-ingress-controller
namespace: default
spec:
replicas: 2
selector:
matchLabels:
k8s-app: nginx-ingress-lb
template:
metadata:
annotations:
prometheus.io/scrape: 'true'
prometheus.io/port: '10254'
labels:
k8s-app: nginx-ingress-lb
spec:
serviceAccountName: nginx-ingress-serviceaccount
containers:
- name: nginx-ingress-controller
image: quay.io/aledbf/nginx-ingress-controller:0.348
args:
- /nginx-ingress-controller
- --default-backend-service=default/default-http-backend
- --default-ssl-certificate=$(POD_NAMESPACE)/tls-certificate
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
ports:
- name: http
containerPort: 80
- name: https
containerPort: 443 |
I think you need |
@gertcuykens no because then nginx is not involved at all |
In my example above the apiVersion: v1
kind: Service
metadata:
name: dam-service
namespace: default
spec:
selector:
k8s-app: dam-app
ports:
- port: 8080
targetPort: 8080
name: http
- port: 5151
targetPort: 5151
name: grpc |
So basically you start a grpc server with the same nginx tls credentials right? And you don't have to do Dial grpc.WithInsecure() on the client because nginx wil take care of internal and external grpc tls correct? Can you confirm the following is correct please?
|
I am using this for the client: import(
...
"crypto/x509"
"google.golang.org/grpc/credentials"
) pool, err := x509.SystemCertPool()
creds := credentials.NewClientTLSFromCert(pool, "")
grpc.Dial(address, grpc.WithTransportCredentials(creds)) However, I have not yet confirmed a successful connection, but I can |
Thx, do you also use server tls credentials on the backend behind the nginx ingress controller? |
AFAIK should not need to. If TLS is terminated at the ingress level, as far as the gRPC service is concerned, it thinks it is running in an insecure environment (which is true, but only inside the protected cluster). If I'm wrong about that somebody pls correct me. |
As long as it's in the cluster it self I prefer performance over security. Just want to be sure that dropping tls doesn't mean switching into something non http2. Don't know enough about grpc if it is always in http2 mode no matter using tls or not. For example my golang web servers drop http2 when tls is disabled |
Adding @stanley-cheung as he is both grpc and nginx expert to maybe answer that question |
PS I suspect this going to be a popular thread can you always update your first post with for example the port change and so on, so others don't need to read the discussion to peace together the grpc example. Thanks. Maybe also strip down duplicate config parts in followup messages. Thanks |
@pcj @gertcuykens would be awesome if any of you are willing to send a PR to add a gRPC example here |
Totally agree, also much cleaner to followup and review |
@aledbf Yes good idea I'll plan to push an example. |
The good news is that this is definitely working for the gRPC service. My go client is using the SSL config above (using the system cert pool) calling grpc.example.com:443, TLS gets terminated, call gets routed to the upstream service (which is not using TLS), and my unary gRPC method is being executed and return grpc status OK. Sweet! I suspect that configuring the grpc server to use TLS should work in conjunction with
spec:
rules:
- host: grpc.example.com
http:
paths:
- path: /
backend:
serviceName: dam-service
servicePort: grpc
The bad news is that I can't seem to get it working when the ingress.yaml also defines a vanilla http ServiceSpec as well (which my initial example here does). I.e. the following does not work: spec:
rules:
- host: grpc.example.com
http:
paths:
- path: /admin
backend:
serviceName: dam-service
servicePort: http
- path: /my.package.MyService
backend:
serviceName: dam-service
servicePort: grpc In this case, the call does seem to get routed to the correct backend service (the non-grpc one at 8080), but the nginx-ingress controller seems to be applying http/2. So, I see this:
Therefore, it appears that although the service selection is correct, grpc / http2 is not. @aledbf is is possible to make the |
Create two Ingress, one adding the grpc annotation to just one |
@aledbf Yea that should work. |
Related to #39 & #2207, I'm psyched to try #2223 + #2246. However, I'm hoping we can use this issue to define an example set of yaml files to use this new feature. I've posted my current set of nginx-ingress config files and hoping to get clarification best practices for gRPC use. I've left out the
rbac.yaml
file as I assume that does not need any changes.I have a gRPC service defined as follows.
dam-app
is an insecure gRPC server listening on TCP port 5151. For simplicity, I'm looking to expose 5151 externally that routes grpc traffic 5151:5151 to the service / deployment. I'm using kube-cert-manager for dynamic letsencrypt certificates and would like to terminate TLS at the nginx-ingress level and run insecure grpc services internally (perhaps I should terminate TLS in the POD? Not sure I need that).Based on PR #2223, looks like all I need to do is:
gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.15
withquay.io/aledbf/nginx-ingress-controller:0.348
.nginx.ingress.kubernetes.io/grpc-backend: "true"
to mygprc-ingress.yaml
.Is that it?
Is a fancier default backend needed?
(any suggestions on tuning up these yaml files also welcome)
One more thing, what is going to be the recommended image going forward?
Thanks for all your work @aledbf and others on this feature!
The text was updated successfully, but these errors were encountered: