步骤12-EFK日志收集.md

12.1 配置工作线程节点的权限

获取工作线程节点Role ARN

STACK_NAME=$(eksctl get nodegroup --cluster eksworkshop -o json | jq -r '.[].StackName')
ROLE_NAME=$(aws cloudformation describe-stack-resources --stack-name $STACK_NAME | jq -r '.StackResources[] | select(.ResourceType=="AWS::IAM::Role") | .PhysicalResourceId')
echo "export ROLE_NAME=${ROLE_NAME}" | tee -a ~/.bash_profile

创建权限Policy文件，主要是CloudWatch Logs权限

cat <<EoF > ./eks-fluent-bit-daemonset-policy.json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "logs:PutLogEvents",
            "Resource": "arn:aws-cn:logs:*:*:log-group:*:*:*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:DescribeLogStreams",
                "logs:PutLogEvents"
            ],
            "Resource": "arn:aws-cn:logs:*:*:log-group:*"
        },
        {
            "Effect": "Allow",
             "Action": [
                "logs:CreateLogGroup",
                "logs:DescribeLogGroups"
            ],
            "Resource": "*"
        }
    ]
}
EoF

为工作线程节点Role增加CloudWatch Logs权限

aws iam put-role-policy --role-name $ROLE_NAME \
--policy-name Logs-Policy-For-Worker \
--policy-document file://./eks-fluent-bit-daemonset-policy.json

查看Policy是否已附加到工作线程节点Role

aws iam get-role-policy --role-name $ROLE_NAME \
--policy-name Logs-Policy-For-Worker

12.2 创建Amazon Elasticsearch Service

使用CLI命令行创建一个包含2节点的Elasticsearch Domain

aws es create-elasticsearch-domain \
  --domain-name kubernetes-logs \
  --elasticsearch-version 7.4 \
  --elasticsearch-cluster-config \
  InstanceType=m5.large.elasticsearch,InstanceCount=2 \
  --ebs-options EBSEnabled=true,VolumeType=standard,VolumeSize=100 \
  --access-policies '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["*"]},"Action":["es:*"],"Resource":"*"}]}'

查看Elasticsearch Domian创建状态

aws es describe-elasticsearch-domain --domain-name kubernetes-logs \
--query 'DomainStatus.Processing'

12.3 部署Fluent-bit

下载Fluent-bit.yml文件

wget https://github.com/aws-samples/eks-workshop-greater-china/blob/master/china/2020_EKS_Launch_Workshop/resource/efk/fluent-bit.yaml

部署Fluent-bit

kubectl apply -f ./fluent-bit.yaml

观察Fluentd Pod状态，直到其处于Running状态

kubectl get pods -w

通过查看日志，验证 Fluent Bit 守护程序集：

kubectl logs ds/fluentbit

参考输出

Found 2 pods, using pod/fluentbit-lq6qb
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
AWS for Fluent Bit Container Image Version 2.3.0
tput: No value for $TERM and no -T specified
Fluent Bit v1.4.2
* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2020/04/12 14:57:09] [ info] [storage] version=1.0.3, initializing...
[2020/04/12 14:57:09] [ info] [storage] in-memory
[2020/04/12 14:57:09] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2020/04/12 14:57:09] [ info] [engine] started (pid=1)
time="2020-04-12T14:57:09Z" level=info msg="[cloudwatch 0] plugin parameter log_group = 'fluent-bit-cloudwatch'\n"
time="2020-04-12T14:57:09Z" level=info msg="[cloudwatch 0] plugin parameter log_stream_prefix = 'from-fluent-bit-'\n"
time="2020-04-12T14:57:09Z" level=info msg="[cloudwatch 0] plugin parameter log_stream = ''\n"
time="2020-04-12T14:57:09Z" level=info msg="[cloudwatch 0] plugin parameter region = 'cn-northwest-1'\n"
time="2020-04-12T14:57:09Z" level=info msg="[cloudwatch 0] plugin parameter log_key = ''\n"
time="2020-04-12T14:57:09Z" level=info msg="[cloudwatch 0] plugin parameter role_arn = ''\n"
time="2020-04-12T14:57:09Z" level=info msg="[cloudwatch 0] plugin parameter auto_create_group = 'true'\n"
time="2020-04-12T14:57:09Z" level=info msg="[cloudwatch 0] plugin parameter endpoint = ''\n"
time="2020-04-12T14:57:09Z" level=info msg="[cloudwatch 0] plugin parameter credentials_endpoint = \n"
time="2020-04-12T14:57:09Z" level=info msg="[cloudwatch 0] plugin parameter log_format = ''\n"
[2020/04/12 14:57:09] [ info] [sp] stream processor started

12.4 将CloudWatch Logs流式传输到Elasticsearch

创建传输过程中使用的Lambda附加的Role:lambda_basic_execution

cat <<EoF > ~/efk/lambda-policy.json
{
   "Version": "2012-10-17",
   "Statement": [
   {
     "Effect": "Allow",
     "Principal": {
        "Service": "lambda.amazonaws.com"
     },
   "Action": "sts:AssumeRole"
   }
 ]
}
EoF
aws iam create-role --role-name lambda_basic_execution --assume-role-policy-document file://~/lambda-policy.json
aws iam attach-role-policy --role-name lambda_basic_execution --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

为lambda_basic_execution增加Elasticsarch相关权限

注意：需要将'Resource'修改为新建的Elasticsearch Domain的ARN

cat <<EoF > ./lambda-es-policy.json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "es:*"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws-cn:es:region:account-id:domain/target-domain-name/*"
        }
    ]
}
EoF
aws iam put-role-policy --role-name lambda_basic_execution --policy-name lambda-es-policy --policy-document file://./lambda-es-policy.json
aws iam get-role-policy --role-name lambda_basic_execution --policy-name lambda-es-policy

登陆AWS Console进行操作，选择log group: fluent-bit-cloudwatch 选择Elasticsearch Cluster kubernetes-logs 和IAM Role lambda_basic_execution 选择Comm log Format 查看所有配置，点击Start Streaming 打开Lambda函数界面，选择函数LogsToElasticsearch_kubernetes-logs，修改正文中 function buildRequest中的var endpointParts

var endpointParts = endpoint.match(/^([^\.]+)\.?([^\.]*)\.?([^\.]*)\.amazonaws\.com$/);
修改为：
var endpointParts = endpoint.match(/^([^\.]+)\.?([^\.]*)\.?([^\.]*)\.amazonaws\.com\.cn$/);

打开Elasticsearch界面，选择kubernetes-logs 打开Kibana URL，几分钟后，ES中将会采集到数据。
将索引规则设置为 cwl-* 下拉菜单中选择*@timestamp*，并创建索引规则 点击发现，以探索日志

12.5 清理环境

kubectl delete -f ./fluentd.yml
aws es delete-elasticsearch-domain --domain-name kubernetes-logs
aws logs delete-log-group --log-group-name /eks/eksworkshop/containers