Skip to content
Permalink
Browse files

Use K8 ingress for TLS termination (#1443)

* Ingress config

The simplest setup ingress is to only proxy HTTP2 traffic.
Multiplexing gRPC and HTTP is possible, but it requires two ingress
objects and explicit path / gRPC service specific forwarding rules.

Notes:
- Supply a default backend to prevent the ingress controller from
  creating it's own
- Supply path routes. Without path routing NGINX won't use our TLS certs
  and will supply it's own "default" TLS cert.

* GCE specific ingress config

- Link to a static IP resource.
- Disable HTTP to slightly simplify the firewall rules.
- Set the backend protocol to HTTP2.
  - Requiries an HTTP2 healthcheck at '/'.
  - Requires TLS (HTTP2 + TLS is incompatible with cmux).
  - Supports GRPC.

Refs
- https://cloud.google.com/load-balancing/docs/https/

* Supply explicit NodePorts

This makes debugging slightly easier since NodePorts will be stable

* Use kustomize build

Using kustomize is required because `kubectl -k` doesn't support
directories to as resources in kustomize file.

* Give baremetal ingress a nodeport

This makes the ingress object routable by Kubernetes in Docker (KIND)
The current kind config routes localhost traffic on 443 to specific node ports (80443)

* Set backend protocol for nginx

* Give custom nodeport it's own patch
  • Loading branch information
gdbelvin committed Feb 8, 2020
1 parent ab27b74 commit 57ba06e2c7cfe94baf7ec7c1e3e2dbc7d91af3a6
Showing with 407 additions and 24 deletions.
  1. +23 −0 deploy/kubernetes/base/ingress.yaml
  2. +1 −1 deploy/kubernetes/base/kustomization.yaml
  3. +3 −1 deploy/kubernetes/base/server-service.yaml
  4. +9 −0 deploy/kubernetes/overlays/gke/ingress.yaml
  5. +4 −1 deploy/kubernetes/overlays/gke/kustomization.yaml
  6. +7 −0 deploy/kubernetes/overlays/gke/managed-cert.yaml
  7. +2 −3 deploy/kubernetes/overlays/gke/server-service.yaml
  8. +21 −0 deploy/kubernetes/overlays/local/ingress-nginx/README.md
  9. +10 −0 deploy/kubernetes/overlays/local/ingress-nginx/baremetal/custom-nodeport.yaml
  10. +7 −0 deploy/kubernetes/overlays/local/ingress-nginx/baremetal/kustomization.yaml
  11. +16 −0 deploy/kubernetes/overlays/local/ingress-nginx/baremetal/service-nodeport.yaml
  12. +74 −0 deploy/kubernetes/overlays/local/ingress-nginx/cloud-generic/deployment.yaml
  13. +52 −0 deploy/kubernetes/overlays/local/ingress-nginx/cloud-generic/kustomization.yaml
  14. +11 −0 deploy/kubernetes/overlays/local/ingress-nginx/cloud-generic/role-binding.yaml
  15. +39 −0 deploy/kubernetes/overlays/local/ingress-nginx/cloud-generic/role.yaml
  16. +4 −0 deploy/kubernetes/overlays/local/ingress-nginx/cloud-generic/service-account.yaml
  17. +16 −0 deploy/kubernetes/overlays/local/ingress-nginx/cloud-generic/service.yaml
  18. 0 deploy/kubernetes/overlays/local/ingress-nginx/kustomization.yaml
  19. +58 −0 deploy/kubernetes/overlays/local/ingress-nginx/static/clusterrole.yaml
  20. +17 −0 deploy/kubernetes/overlays/local/ingress-nginx/static/clusterrolebinding.yaml
  21. +4 −0 deploy/kubernetes/overlays/local/ingress-nginx/static/kustomization.yaml
  22. +16 −0 deploy/kubernetes/overlays/local/ingress-nginx/static/limitrange.yaml
  23. +9 −0 deploy/kubernetes/overlays/local/ingress-nginx/static/namespace.yaml
  24. +2 −2 deploy/kubernetes/overlays/local/kustomization.yaml
  25. +0 −14 deploy/kubernetes/overlays/local/server-service.yaml
  26. +2 −2 scripts/kubernetes_test.sh
@@ -0,0 +1,23 @@
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: kt-ingress
annotations:
nginx.ingress.kubernetes.io/backend-protocol: "GRPCS"
spec:
backend:
serviceName: server
servicePort: grpc-api
tls:
- hosts:
- "localhost"
- "127.0.0.1"
- "sandbox.keytransparency.dev"
secretName: kt-tls
rules:
- http:
paths:
- path: /
backend:
serviceName: server
servicePort: grpc-api
@@ -4,6 +4,7 @@ commonLabels:
resources:
- db-deployment.yaml
- db-service.yaml
- ingress.yaml
- init-pod.yaml
- log-server-deployment.yaml
- log-server-service.yaml
@@ -19,4 +20,3 @@ resources:
- sequencer-service.yaml
- server-deployment.yaml
- server-service.yaml

@@ -9,11 +9,13 @@ spec:
- name: "grpc-api"
port: 443
targetPort: 8080
nodePort: 30080
- name: "http-metrics"
port: 8081
targetPort: 8081
nodePort: 30081
selector:
io.kompose.service: server
type: NodePort
type: NodePort # Required for ingress
status:
loadBalancer: {}
@@ -0,0 +1,9 @@
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: kt-ingress
annotations:
kubernetes.io/ingress.class: "gce"
kubernetes.io/ingress.global-static-ip-name: "kt-ingress-ip"
kubernetes.io/ingress.allow-http: "false"
networking.gke.io/managed-certificates: sandbox-keytransparency-dev
@@ -1,11 +1,14 @@
bases:
- ../../base
resources:
- managed-cert.yaml
patchesStrategicMerge:
- ingress.yaml
- log-server-stackdriver-prometheus-sidecar.yaml
- log-signer-stackdriver-prometheus-sidecar.yaml
- map-server-stackdriver-prometheus-sidecar.yaml
- sequencer-stackdriver-prometheus-sidecar.yaml
- server-stackdriver-prometheus-sidecar.yaml
- server-service.yaml
- server-stackdriver-prometheus-sidecar.yaml


@@ -0,0 +1,7 @@
apiVersion: networking.gke.io/v1beta1
kind: ManagedCertificate
metadata:
name: sandbox-keytransparency-dev
spec:
domains:
- sandbox.keytransparency.dev
@@ -2,6 +2,5 @@ apiVersion: v1
kind: Service
metadata:
name: server
spec:
type: LoadBalancer

annotations:
cloud.google.com/app-protocols: '{"grpc-api":"HTTP2"}'
@@ -0,0 +1,21 @@
# NGINX Configs

Installing baremetal NGINX requires running the following commands according to the [directions](https://kubernetes.github.io/ingress-nginx/deploy/) on the nginx site.

```
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/mandatory.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/cloud-generic.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/baremetal/service-nodeport.yaml
```

The directories below contain the contents of these three configs
1. Split out into their component yaml files.
2. Added `kustomization.yaml` files tying them together.
3. Removed conflicting resources.

The kustomize dependency graph looks like so:
```
overlays\local -> overlays\local\ingress-nginx\baremetal
overlays\local\ingress-nginx\baremetal -> overlays\local\ingress-nginx\cloudgeneric
overlays\local\ingress-nginx\cloudgeneric -> overlays\local\ingress-nginx\static
```
@@ -0,0 +1,10 @@
apiVersion: v1
kind: Service
metadata:
name: ingress-nginx
spec:
type: NodePort
ports:
- name: https
port: 443
nodePort: 30443
@@ -0,0 +1,7 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
- ../cloud-generic
patchesStrategicMerge:
- custom-nodeport.yaml
- service-nodeport.yaml
@@ -0,0 +1,16 @@
apiVersion: v1
kind: Service
metadata:
name: ingress-nginx
spec:
type: NodePort
ports:
- name: http
port: 80
targetPort: 80
protocol: TCP
- name: https
port: 443
targetPort: 443
protocol: TCP
externalTrafficPolicy: Cluster
@@ -0,0 +1,74 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-ingress-controller
spec:
replicas: 1
template:
metadata:
annotations:
prometheus.io/port: "10254"
prometheus.io/scrape: "true"
spec:
# wait up to five minutes for the drain of connections
terminationGracePeriodSeconds: 300
serviceAccountName: nginx-ingress-serviceaccount
containers:
- name: nginx-ingress-controller
image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:master
args:
- /nginx-ingress-controller
- --configmap=$(POD_NAMESPACE)/$(NGINX_CONFIGMAP_NAME)
- --tcp-services-configmap=$(POD_NAMESPACE)/$(TCP_CONFIGMAP_NAME)
- --udp-services-configmap=$(POD_NAMESPACE)/$(UDP_CONFIGMAP_NAME)
- --publish-service=$(POD_NAMESPACE)/$(SERVICE_NAME)
- --annotations-prefix=nginx.ingress.kubernetes.io
securityContext:
allowPrivilegeEscalation: true
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
# www-data -> 101
runAsUser: 101
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
ports:
- name: http
containerPort: 80
protocol: TCP
- name: https
containerPort: 443
protocol: TCP
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 10
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 10
lifecycle:
preStop:
exec:
command:
- /wait-shutdown
@@ -0,0 +1,52 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
- ../static
namespace: ingress-nginx
commonLabels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
resources:
- deployment.yaml
- role-binding.yaml
- role.yaml
- service-account.yaml
- service.yaml
images:
- name: quay.io/kubernetes-ingress-controller/nginx-ingress-controller
newTag: master
vars:
- fieldref:
fieldPath: metadata.name
name: NGINX_CONFIGMAP_NAME
objref:
apiVersion: v1
kind: ConfigMap
name: nginx-configuration
- fieldref:
fieldPath: metadata.name
name: TCP_CONFIGMAP_NAME
objref:
apiVersion: v1
kind: ConfigMap
name: tcp-services
- fieldref:
fieldPath: metadata.name
name: UDP_CONFIGMAP_NAME
objref:
apiVersion: v1
kind: ConfigMap
name: udp-services
- fieldref:
fieldPath: metadata.name
name: SERVICE_NAME
objref:
apiVersion: v1
kind: Service
name: ingress-nginx
configMapGenerator:
- name: nginx-configuration
- name: tcp-services
- name: udp-services
generatorOptions:
disableNameSuffixHash: true
@@ -0,0 +1,11 @@
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: nginx-ingress-role-nisa-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: nginx-ingress-role
subjects:
- kind: ServiceAccount
name: nginx-ingress-serviceaccount
@@ -0,0 +1,39 @@
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
name: nginx-ingress-role
rules:
- apiGroups:
- ""
resources:
- configmaps
- pods
- secrets
- namespaces
verbs:
- get
- apiGroups:
- ""
resources:
- configmaps
resourceNames:
# Defaults to "<election-id>-<ingress-class>"
# Here: "<ingress-controller-leader>-<nginx>"
# This has to be adapted if you change either parameter
# when launching the nginx-ingress-controller.
- "ingress-controller-leader-nginx"
verbs:
- get
- update
- apiGroups:
- ""
resources:
- configmaps
verbs:
- create
- apiGroups:
- ""
resources:
- endpoints
verbs:
- get
@@ -0,0 +1,4 @@
apiVersion: v1
kind: ServiceAccount
metadata:
name: nginx-ingress-serviceaccount
@@ -0,0 +1,16 @@
kind: Service
apiVersion: v1
metadata:
name: ingress-nginx
spec:
externalTrafficPolicy: Local
type: LoadBalancer
ports:
- name: http
port: 80
protocol: TCP
targetPort: http
- name: https
port: 443
protocol: TCP
targetPort: https
No changes.

0 comments on commit 57ba06e

Please sign in to comment.
You can’t perform that action at this time.