fluent-bit-appmesh

Fluent Bit and AWS App Mesh

In this example, we will showcase how to parse the Envoy access logs for AWS App Mesh in an Amazon EKS cluster. To achieve this, we will deploy Fluent Bit as a DaemonSet to tail /var/log/containers/*.log on the EKS cluster and forward to CloudWatch.

Prerequisites

• An EKS Cluster with the appropriate IAM Roles attached to the worker nodes for App Mesh and CloudWatch. This can easily be achieved by standing up a cluster with eksctl and adding the appropriate IAM add-on policies.

• Enable App Mesh Integration on the cluster.

• Set up the Color App as described in the AWS App Mesh Documentation. We will edit deployment below.

Instructions

• To get started, create the Namespace that's needed for the the Fluent Bit components.

kubectl apply -f logging-ns.yaml

• Deploy the Fluent Bit DaemonSet.

kubectl apply -f fluent-bit/

• Deploy the Color App.

1. There is a file called color.yaml which is the same manifest that's used in the Color App from the AWS App Mesh Documentation in the prerequisites. However, it's been slightly modified to emit the colorteller-black Envoy access logs to /dev/stdout and also adds an annotation of fluentbit.io/parser: envoy to the colorteller-black container. This annotation suggests that the data should be processed using the pre-defined parser called envoy which is defined in the fluent-bit-configmap.yaml. This configuration is how we will wire up Fluent Bit to parse the Envoy access logs for App Mesh. Please see this link for more info on pre-defined parsers in Fluent Bit.

kubectl apply -f color-app/color.yaml

2. You should already have a curler container if you followed the Color App instructions from the prerequisites. Exec into the curler container and rerun the below:

for i in {1..100}; do curl colorgateway:9080/color; echo; done

3. View the colorteller-black logs in CloudWatch. The Envoy access logs should now be parsed like the example below:

{
    "kubernetes": {
        "annotations": {
            "fluentbit.io/parser": "envoy",
            "kubernetes.io/psp": "eks.privileged"
        },
        "container_hash": "b10687cb4b94ef7aecc0c6e815efb56c8d8889db5316bafc42477acd908a0e91",
        "container_name": "envoy",
        "docker_id": "3be97d47d717ae3ba9937d1bd58b683cdb8b24bd5c66cf7fefeb2ee47c808b08",
        "host": "ip-192-168-10-112.ec2.internal",
        "labels": {
            "app": "colorteller",
            "pod-template-hash": "d868b5bc9",
            "version": "black"
        },
        "namespace_name": "appmesh-demo",
        "pod_id": "42336070-4796-11ea-ac15-0278ee4c2031",
        "pod_name": "colorteller-black-d868b5bc9-zwz28"
    },
    "log": "[2020-02-05T16:37:27.958Z] \"GET / HTTP/1.1\" 200 - 0 5 0 0 \"-\" \"Go-http-client/1.1\" \"6c6ae4b8-60ac-98b6-ba32-a3a4a5c938bf\" \"colorteller.appmesh-demo:9080\" \"127.0.0.1:9080\"\n",
    "log_processed": {
        "authority": "colorteller.appmesh-demo:9080",
        "bytes_received": "0",
        "bytes_sent": "5",
        "code": "200",
        "duration": "0",
        "method": "GET",
        "path": "/",
        "protocol": "HTTP/1.1",
        "request_id": "6c6ae4b8-60ac-98b6-ba32-a3a4a5c938bf",
        "response_flags": "-",
        "start_time": "2020-02-05T16:37:27.958Z",
        "upstream_host": "127.0.0.1:9080",
        "user_agent": "Go-http-client/1.1",
        "x_envoy_upstream_service_time": "0",
        "x_forwarded_for": "-"
    },
    "stream": "stdout",
    "time": "2020-02-05T16:37:33.976469719Z"
}