Application provisioning on an Amazon EKS cluster with ingress and monitoring capabilities using Terraform and Helm
| Name | Version |
|---|---|
| terraform | ~> 1.2.2 |
| eks | 1.22 |
| kubectl | v1.24.2 |
| aws-cli | 2.7.10 |
| ingress-nginx helm chart | 4.1.4 |
| kube-prometheus-stack helm chart | 36.6.1 |
| kind node image | kindest/node:v1.22.9 |
| go | 1.18 |
The application that will be deployed, which resides in the /app directory, is a simple Go web application that scrapes the Kubernetes API server and shows a dashboard with information about all running pods in its namespace.
To provision infrastructure in the AWS cloud, the aws-cli tool is needed, in order to configure your account's credentials. Alternatively, store them in two terraform variables named AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY and use them in all terraform directories to configure the aws provider. It is important that the latest version AWS CLI version 2 is used, otherwise in following steps, the kubeconfig file comes with an error - it uses the invalid apiVersion: client.authentication.k8s.io/v1alpha1 instead of the correct apiVersion: client.authentication.k8s.io/v1beta1.
For extra safety and isolation, the EKS cluster, the ingress-nginx controller and the kube-prometheus-stack are all managed by different directories. To further experiment and explore the terraform EKS provisioning procedure, the official EKS module was not used - instead, the EKS cluster was build with simple terraform resources (e.g. aws_eks_cluster, aws_eks_node_group, etc.).
Firstly, navigate to the /eks directory and deploy the eks cluster. Upon completion, the eks service will be deployed, as well as two node groups, the monitoring and the application groups. Both have a group label, but the first one has a group=monitoring:NoSchedule taint as well, so just the pods created by the kube-prometheus-stack helm chart will be scheduled only on these nodes. The cluster configuration will be included in the ${HOME}/.kube/config file and will be updated every time the eks cluster changes.
To verify the correct cluster status, run for example:
kubectl get nodes --show-labelsThen, in the /ingress-nginx directory, deploy the ingress-nginx controller using Helm, specifying just two chart values regarding the controller update strategy.
After making sure that the ingress controller is in a stable state (for example, to avoid a failed calling webhook "validate.nginx.ingress.kubernetes.io" error), deploy the kube-prometheus-stack from the /monitoring directory, again using Helm. The file /monitoring/values.yaml contains values that create ingresses for the prometheus, grafana and alertmanager components and supply them with the node affinity rules and tolerations needed for the monitoring node group.
Lastly, the /app-manifests directory is responsible for the application deployment. Along with the usual configuration, an ingress resource is created in order for the app to be accessible from the /app route, as well as a ServiceMonitor resource for the app to be visible and monitored by the monitoring stack.
To find the hostname of the AWS ELB provisioned by the ingress controller, use the kubectl tool and execute the following command:
kubectl get svc -n ingress-nginx ingress-nginx-controller -o jsonpath='{.status.loadBalancer.ingress[0].hostname}{"\n"}'Alternatively, this hostname can also be found in the AWS Console in the EC2 Load balancers section.
After applying all of the above steps, the following routes are exposed:
| Name | Type |
|---|---|
| /app | Prefix |
| /monitor/prometheus | Prefix |
| /monitor/grafana | Prefix |
| /monitor/alertmanager | Prefix |
To setup a local K8S cluster that mimics the one that would be deployed on AWS, navigate to the /kind directory
and use the configuration to deploy a kind cluster. For all the other directories, the only change needed is to set the variable env equal to kind (instead of the default aws). Everything else stays the same and all routes will ultimately be hosted at localhost.
Copyright © 2022 Kyriakos Tsaliagkos
This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program. If not, see https://www.gnu.org/licenses/.