Skip to content

Commit

Permalink
Add example manifests for the NGINX Ingress controller
Browse files Browse the repository at this point in the history
  • Loading branch information
@anton-johansson
anton-johansson committed Aug 19, 2019
1 parent 97c5f3e commit 5df7f17
Show file tree
Hide file tree
Showing 5 changed files with 377 additions and 0 deletions.
16 changes: 16 additions & 0 deletions services/1-mandatory/4-nginx-ingress-controller/1-namespace.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
---
apiVersion: v1
kind: Namespace
metadata:
name: ingress-nginx

---
kind: Secret
apiVersion: v1
metadata:
name: default-server-secret
namespace: ingress-nginx
type: Opaque
data:
tls.crt: <base64-encoded certificate>
tls.key: <base64-encoded private key>
96 changes: 96 additions & 0 deletions services/1-mandatory/4-nginx-ingress-controller/2-rbac.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: nginx-ingress-controller
namespace: ingress-nginx

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: nginx-ingress-controller
rules:
- apiGroups: ['']
resources: [configmaps, endpoints, nodes, pods, secrets]
verbs: [list, watch]
- apiGroups: ['']
resources: [nodes]
verbs: [get]
- apiGroups: ['']
resources: [services]
verbs: [get, list, watch]
- apiGroups: ['']
resources: [events]
verbs: [create, patch]
- apiGroups: [extensions, networking.k8s.io]
resources: [ingresses]
verbs: [get, list, watch]
- apiGroups: [extensions, networking.k8s.io]
resources: [ingresses/status]
verbs: [update]

---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: nginx-ingress-controller
namespace: ingress-nginx
rules:
- apiGroups: ['']
resources: [configmaps, pods, secrets, namespaces]
verbs: [get]
- apiGroups: ['']
resources: [configmaps]
resourceNames: [ingress-controller-leader-external, ingress-controller-leader-internal]
verbs: [get, update]
- apiGroups: ['']
resources: [configmaps]
verbs: [create]
- apiGroups: ['']
resources: [endpoints]
verbs: [get]

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: psp:nginx-ingress-controller
roleRef:
kind: ClusterRole
name: psp:privileged
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: nginx-ingress-controller
namespace: ingress-nginx

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: nginx-ingress-controller
roleRef:
kind: ClusterRole
apiGroup: rbac.authorization.k8s.io
name: nginx-ingress-controller
subjects:
- kind: ServiceAccount
name: nginx-ingress-controller
namespace: ingress-nginx

---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: nginx-ingress-controller
namespace: ingress-nginx
roleRef:
kind: Role
apiGroup: rbac.authorization.k8s.io
name: nginx-ingress-controller
subjects:
- kind: ServiceAccount
name: nginx-ingress-controller
namespace: ingress-nginx

113 changes: 113 additions & 0 deletions services/1-mandatory/4-nginx-ingress-controller/3-daemonset-external.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
---
kind: ConfigMap
apiVersion: v1
metadata:
name: nginx-configuration-external
namespace: ingress-nginx
data:
log-format-escape-json: 'true'
log-format-upstream: '{"time":"$time_iso8601","accessType":"request","accessClass":"external","remote_addr":"$remote_addr","request_id":"$req_id","remote_user":"$remote_user","bytes_sent":$bytes_sent,"request_time":$request_time,"status":"$status","host":"$host","request_protocol":"$server_protocol","path":"$uri","query_string":"$args","request_length":$request_length,"duration":$request_time,"method":"$request_method","referrer": "$http_referer","user_agent":"$http_user_agent","upstream_name":"$proxy_upstream_name","upstream_addr":"$upstream_addr","geo":{"country":"$geoip_country_code","city":"$geoip_city"}}'
log-format-stream: 'escape=json ''{"time":"$time_iso8601","accessType":"stream","accessClass":"external","remote_addr":"$remote_addr","bytes_sent":$bytes_sent,"status":"$status","request_protocol":"$protocol","session_time":$session_time,"upstream_name":"$proxy_upstream_name","upstream_addr":"$upstream_addr"}'''
server-tokens: 'false'

---
kind: ConfigMap
apiVersion: v1
metadata:
name: tcp-services-external
namespace: ingress-nginx
data:

---
kind: ConfigMap
apiVersion: v1
metadata:
name: udp-services-external
namespace: ingress-nginx
data:

---
kind: DaemonSet
apiVersion: apps/v1
metadata:
name: nginx-ingress-controller-external
namespace: ingress-nginx
labels:
app.kubernetes.io/name: nginx-ingress-controller
app.kubernetes.io/component: external
spec:
selector:
matchLabels:
app.kubernetes.io/name: nginx-ingress-controller
app.kubernetes.io/component: external
updateStrategy:
# When we update the NGINX DaemonSet, we want to do it in a controlled manner.
# See the README for more information
type: OnDelete
template:
metadata:
labels:
app.kubernetes.io/name: nginx-ingress-controller
app.kubernetes.io/component: external
annotations:
prometheus.io/scrape: 'true'
prometheus.io/path: /metrics
prometheus.io/port: '10254'
spec:
serviceAccountName: nginx-ingress-controller
nodeSelector:
kubernetes.io/role: nginx
containers:
- name: nginx-ingress-controller
image: 'quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.24.1'
ports:
- name: http
containerPort: 80
hostPort: 80
hostIP: '10.0.0.110,10.0.0.110'
- name: https
containerPort: 443
hostPort: 443
hostIP: '10.0.0.110,10.0.0.111'
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
securityContext:
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
args:
- '/nginx-ingress-controller'
- '--ingress-class=external'
- '--configmap=$(POD_NAMESPACE)/nginx-configuration-external'
- '--tcp-services-configmap=$(POD_NAMESPACE)/tcp-services-external'
- '--udp-services-configmap=$(POD_NAMESPACE)/udp-services-external'
- '--default-ssl-certificate=$(POD_NAMESPACE)/default-server-secret'
- '--enable-ssl-chain-completion=false'
113 changes: 113 additions & 0 deletions services/1-mandatory/4-nginx-ingress-controller/4-daemonset-internal.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
---
kind: ConfigMap
apiVersion: v1
metadata:
name: nginx-configuration-internal
namespace: ingress-nginx
data:
log-format-escape-json: 'true'
log-format-upstream: '{"time":"$time_iso8601","accessType":"request","accessClass":"internal","remote_addr":"$remote_addr","request_id":"$req_id","remote_user":"$remote_user","bytes_sent":$bytes_sent,"request_time":$request_time,"status":"$status","host":"$host","request_protocol":"$server_protocol","path":"$uri","query_string":"$args","request_length":$request_length,"duration":$request_time,"method":"$request_method","referrer": "$http_referer","user_agent":"$http_user_agent","upstream_name":"$proxy_upstream_name","upstream_addr":"$upstream_addr","geo":{"country":"$geoip_country_code","city":"$geoip_city"}}'
log-format-stream: 'escape=json ''{"time":"$time_iso8601","accessType":"stream","accessClass":"internal","remote_addr":"$remote_addr","bytes_sent":$bytes_sent,"status":"$status","request_protocol":"$protocol","session_time":$session_time,"upstream_name":"$proxy_upstream_name","upstream_addr":"$upstream_addr"}'''
server-tokens: 'false'

---
kind: ConfigMap
apiVersion: v1
metadata:
name: tcp-services-internal
namespace: ingress-nginx
data:

---
kind: ConfigMap
apiVersion: v1
metadata:
name: udp-services-internal
namespace: ingress-nginx
data:

---
kind: DaemonSet
apiVersion: apps/v1
metadata:
name: nginx-ingress-controller-internal
namespace: ingress-nginx
labels:
app.kubernetes.io/name: nginx-ingress-controller
app.kubernetes.io/component: internal
spec:
selector:
matchLabels:
app.kubernetes.io/name: nginx-ingress-controller
app.kubernetes.io/component: internal
updateStrategy:
# When we update the NGINX DaemonSet, we want to do it in a controlled manner.
# See the README for more information
type: OnDelete
template:
metadata:
labels:
app.kubernetes.io/name: nginx-ingress-controller
app.kubernetes.io/component: internal
annotations:
prometheus.io/scrape: 'true'
prometheus.io/path: /metrics
prometheus.io/port: '10254'
spec:
serviceAccountName: nginx-ingress-controller
nodeSelector:
kubernetes.io/role: nginx
containers:
- name: nginx-ingress-controller
image: 'quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.24.1'
ports:
- name: http
containerPort: 80
hostPort: 80
hostIP: '10.0.0.210,10.0.0.210'
- name: https
containerPort: 443
hostPort: 443
hostIP: '10.0.0.210,10.0.0.211'
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
securityContext:
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
args:
- '/nginx-ingress-controller'
- '--ingress-class=internal'
- '--configmap=$(POD_NAMESPACE)/nginx-configuration-internal'
- '--tcp-services-configmap=$(POD_NAMESPACE)/tcp-services-internal'
- '--udp-services-configmap=$(POD_NAMESPACE)/udp-services-internal'
- '--default-ssl-certificate=$(POD_NAMESPACE)/default-server-secret'
- '--enable-ssl-chain-completion=false'
39 changes: 39 additions & 0 deletions services/1-mandatory/4-nginx-ingress-controller/README.md
View file
Original file line number Diff line number Diff line change
@@ -1 +1,40 @@
# NGINX ingress controller

We use NGINX ([Kubernetes own one](https://github.com/kubernetes/ingress-nginx), not the [NGINX Inc. one](https://github.com/nginxinc/kubernetes-ingress) as our [Ingress controller](https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/). The Ingress controller is required to actually handle `Ingress` resources within the clusters. The NGINX ingress controller finds all `Ingress` resources and builds an `nginx.conf` from that. The idea is to route all HTTP traffic to nodes which have this controller and the controller will in turn route traffic to the appropriate pods.


## Controlling network traffic

To be able to set up proper domain records, we need to know which nodes that will actually run the NGINX controller. If we scale up with more nodes, we don't want to bother with changing the DNS-records. To know which nodes that will actually run the controller, we will label two nodes with `kubernetes.io/role=nginx` by using the following command:

```shell
$ kubectl label node <nodeName> kubernetes.io/role=nginx
```

Then we will utilize a `DaemonSet` resource with a node selector rule. This will cause the `DaemonSet` to place one pod on each of the labelled nodes.


## Internal vs external traffic

For proper isolation, we want to route internal traffic to one Ingress controller and external traffic to another Ingress controller. This is better for security. We will not have to enforce IP whitelists on the internal services (which is easy to forget, causing exposure of internal services) and it's also lets us route traffic on two different virtual IP addresses and we can make it technically impossible to reach internal services from the Internet. External services will have public DNS records pointing to the IP addresses of the external Ingress controllers, while internal services will have **internal DNS records** pointing to the IP addresses of the internal Ingress controllers. This way, there will be no way to access the internal services from the Internet.


## High availability

We want to be able to perform maintenance work on the nodes that hosts the Ingress controllers. For this we need a tool that manages our virtual IP addresses. Each Ingress node will have `keepalived` installed on them, that generates two virtual IP addresses. We will configure them as Active/Active, causing them to have eachother as backups. So if we take one node down, `keepalived` on the other node will take over the first nodes IP addresses. We can then point our DNS records to both of these IP addresses.


## Install

```shell
$ kubectl apply -f .
```

The example manifest above expects two labelled nodes for Ingress with the following IP addresses:

| Node | IP address | Virtual IP address for external traffic | Virtual IP address for internal traffic |
| -------------- | ----------- | --------------------------------------- | --------------------------------------- |
| `k8s-nginx-10` | `10.0.0.10` | `10.0.0.110` | `10.0.0.210` |
| `k8s-nginx-11` | `10.0.0.11` | `10.0.0.111` | `10.0.0.211` |

These IP addresses can of course be changed to any IP of your liking, as long as you use two separate IP addresses for internal and external.

0 comments on commit 5df7f17

Please sign in to comment.