Skip to content

Files

Latest commit

 

History

History

adot-eks-on-ec2-to-cw

README.md

ADOT Helm chart for EKS on EC2 metrics and logs to CW Container Insights

License

The repository contains a Helm chart to provide easy to operate, end-to-end AWS Elastic Kubernetes Service (EKS) on AWS Elastic Compute Cloud (EC2) monitoring with AWS Distro for OpenTelemetry(ADOT) collector for metrics and Fluent Bit for logs. Therefore, this Helm chart is useful for customers who use EKS on EC2 and want to collect metrics and logs to send to Amazon CloudWatch Container Insights.

The Helm chart configured in this repository deploys ADOT Collector and Fluent Bit as DaemonSets and is ready to collect metrics and logs and send them to Amazon CloudWatch Container Insights.

Helm Chart Structure

adot-eks-on-ec2-to-cw/
|-- scripts/ 
|   |-- install-tools.sh
|   |-- lint-charts.sh
|   |-- validate-charts.sh
|-- templates/
|   |-- NOTES.txt
|   |-- aws-for-fluent-bit/
|   |   |-- _helpers.tpl
|   |   |-- clusterrole.yaml
|   |   |-- clusterrolebinding.yaml
|   |   |-- configmap.yaml
|   |   |-- daemonset.yaml
|   |   |-- namespace.yaml
|   |   |-- serviceaccount.yaml
|   |-- aws-otel-collector/
|   |   |-- _helpers.tpl
|   |   |-- clusterrole.yaml
|   |   |-- clusterrolebinding.yaml
|   |   |-- configmap.yaml
|   |   |-- daemonset.yaml
|   |   |-- namespace.yaml
|   |   |-- serviceaccount.yaml
|   |   |-- sidecar.yaml
|   |   |-- sidecarnamespace.yaml
|-- Chart.yaml
|-- values.schema.json
|-- values.yaml

templates folder contains two subfolders, aws-for-fluent-bit and aws-otel-collector, and each subfolder contains template files that will be evaluated with the default values configured in values.yaml.

script folder contains shell script files to run chart validation and lint tests with Helm Lint and Kubeval.

values.yaml file stores parameterized template defaults in the Helm chart. Using this file, we can provide more flexibility to our users to expose configuration that can be overriden at installation and upgrade time.

values.schema.json file contains schemas of each values in values.yaml. It defines each values’ type, required keys, and constraints.

_helpers.tpl files are used to define GO template helpers to create name variables.

Prerequisite

The following prerequisites need to be set up in order to install this Helm chart.

Get Repository Information

Helm must be installed to use the chart. Please refer to Helm's documentation to get started.

Once Helm is set up properly, add this repo as follows:

$ helm repo add [REPO_NAME] https://TO_BE_RELEASED.github.io/adot-helm-eks-ec2
$ helm search repo [REPO_NAME] # Run this command in order to see the charts.

Install Chart

$ helm install \
  [RELEASE_NAME] [REPO_NAME]/adot-eks-on-ec2-to-cw \
  --set clusterName=[CLUSTER_NAME] --set awsRegion=[AWS_REGION]

CLUSTER_NAME and AWS_REGION must be specified with your own EKS cluster and the region. You can find these values by executing following command.

$ kubectl config current-context

[IAM_User_Name]@[CLUSTER_NAME].[AWS_REGION].eksctl.io

To verify the installation is successful, you can execute the following command.

$ kubectl get pods --all-namespaces

NAMESPACE                NAME                             READY   STATUS    RESTARTS   AGE
amazon-cloudwatch        fluent-bit-f27cz                 1/1     Running   0          4s
amazon-cloudwatch        fluent-bit-m2mkr                 1/1     Running   0          4s
amzn-cloudwatch-metrics  adot-collector-daemonset-7nrst   1/1     Running   0          4s
amzn-cloudwatch-metrics  adot-collector-daemonset-x7n8x   1/1     Running   0          4s

If you see these four running pods, two for Fluent Bit and two for ADOT Collector as DaemonSets within the specified namespaces, they are successfully deployed.

Verify the Helm chart works as expected

  • Run chart validation test and lint fromMakeFile.
$ cd adot-eks-on-ec2-to-cw
$ make install-tools # required initially
$ make all           # to run chart validation test and lint

Verify the metrics and logs are sent to Amazon CloudWatch

  • Open Amazon CloudWatch console
  • Select "Logs -> Log groups" on the left navigation bar.
  • Check if following four log groups exist (performance log group will take longer than others).
/aws/containerinsights/[CLUSTER_NAME]/application
/aws/containerinsights/[CLUSTER_NAME]/dataplane
/aws/containerinsights/[CLUSTER_NAME]/host
/aws/containerinsights/[CLUSTER_NAME]/performance
  • Select "Insights -> Container Insights" on the left navigation bar.
  • Choose Performance monitoring in the drop-down menu on the top-left side.
  • Choose the levels such as EKS pods, EKS nodes, and EKS namespaces from the drop-down menu in the automated dashboard.
  • If you observe metrics of the running pods for CPU Utilization, Memory Utilization, etc, the metrics are successfully collected and visualized in Container Insights.

CWCI_dashboard

Configuration

To see all configurable options with detailed comments:

$ helm show values [REPO_NAME]/adot-eks-on-ec2-to-cw

By changing values in values.yaml, you are able to customize the chart to use your preferred configuration.

Following options are some useful configurations that can be applied to this Helm chart.

Deploy ADOT Collector as Sidecar

Sidecar is a microservice design pattern where a companion service runs next to your primary microservice, augmenting its abilities or intercepting resources it is utilizing. The sidecar pattern would be the best fit for a single application monitoring. In order to deploy the ADOT Collector in Sidecar mode using the Helm chart, 1) update sidecar.yaml and values.yaml files in the Helm chart with the application configurations and 2) include the use of --set flag in the helm install command from Install Chart.

$ helm install \
  [RELEASE_NAME] [REPO_NAME]/adot-eks-on-ec2-to-cw \
  --set clusterName=[CLUSTER_NAME] --set awsRegion=[AWS_REGION] \
  --set adotCollector.daemonSet.enabled=false --set adotCollector.sidecar.enabled=true

The use of --set flag with enabled=true or enabled=false can switch on/off the specified deployment mode. The command set enabled=false for ADOT Collector as DaemonSet and enabled=true to deploy ADOT Collector as Sidecar. You can also check whether your applications are successfully deployed by executing the following command.

$ kubectl get pods --all-namespaces

NAMESPACE                NAME                            READY   STATUS    RESTARTS   AGE
adot-sidecar-namespace   adot-sidecar-658dc9ffbb-w9zv2   2/2     Running   0          5m18s
amazon-cloudwatch        fluent-bit-9dcql                1/1     Running   0          5m18s
amazon-cloudwatch        fluent-bit-wqhmd                1/1     Running   0          5m18s

Deploy ADOT Collector as Deployment and StatefulSet

Deploying ADOT Collector as Deployment and StatefulSet mode requires installing ADOT Operator. See OpenTelemetry Operator Helm Chart for detailed explanation.

Deploy ADOT Collector with Prometheus Receiver for AWS Container Insights on EKS

Please refer to deployment template to deploy ADOT Collector with Prometheus Receiver and Amazon CloudWatch Embedded Metric Format (EMF) Exporter for AWS Container Insights on EKS via configurations in the Helm chart.

AWS EKS on Fargate

The prerequisites for Fargate logging via Amazon EKS on AWS Fargate include: 1) Create a Fargate profile for your cluster and 2) Create a Fargate pod execution role.

Amazon EKS on Fargate features a Fluent Bit based built-in log router to send collected logs to various destinations, including Amazon CloudWatch. Fargate utilizes AWS for Fluent Bit, and the required configurations for Fargate to automatically detect and configure the log router are included in the Helm chart in configmap.yaml and values.yaml files based on the Fargate logging user guide. The configurations in configmap.yaml must include the name: aws-logging and the namespace: aws-observability for Fargate logging. To deploy your application to Amazon EKS on Fargate, you need to include your application yaml file in the aws-fargate-logging folder of the Helm chart with the same namespace as your AWS Fargate profile. For more detailed information about Fargate logging, such as deployment of a sample-app.yaml or your application and the instructions to download, create, and attach IAM policy to the pod execution role for Fargate profile, please refer to the user guide for Fargate logging and Getting started with AWS Fargate using Amazon EKS.

This is an example of using the Helm chart for Fargate logging with the sample-app.yaml from Fargate logging.

$ helm install \
  [RELEASE_NAME] [REPO_NAME]/adot-eks-on-ec2-to-cw \
  --set clusterName=[CLUSTER_NAME] --set awsRegion=[AWS_REGION] \
  --set fargateLogging.enabled=true

To confirm the sample-app is deployed and troubleshoot the logging is enabled/disabled, you can run the following commands.

$ kubectl get pods --all-namespaces

NAMESPACE               NAME                            READY   STATUS    RESTARTS   AGE
aws-observability       sample-app-86b8cc866b-cr5x6     1/1     Running   0          13m
aws-observability       sample-app-86b8cc866b-q75z7     1/1     Running   0          13m
aws-observability       sample-app-86b8cc866b-t615c     1/1     Running   0          13m
$ kubectl describe po -n aws-observability sample-app-86b8cc866b-cr5x6

Events:
  Type    Reason          Age  From               Message 
  ----    ------          ---  ----               -------
  Normal  LoggingEnabled  13m  fargate-scheduler  Successfully enabled logging for pod

Uninstall Chart

The following command uninstalls the chart. This will remove all the Kubernetes components associated with the chart and deletes the release.

$ helm uninstall [RELEASE_NAME]

Upgrade Chart

$ helm upgrade [RELEASE_NAME] [REPO_NAME]/adot-eks-on-ec2-to-cw

Contributing

See CONTRIBUTING.md.

Contributors

Hyunuk Lim

James Park

Further Information

Set up Fluent Bit as a DaemonSet to send logs to CloudWatch Logs

Using AWS Distro for OpenTelemetry

License

Apache 2.0 License.

Support Plan

Our team plans to fully support the code we plan to release in this repo.