Failed to source credential on Amazon EKS IAM Roles for Service Account

[yantk-hk]

Upon upgrade to aws-for-fluent 2.8 (fluent bit 1.6)
Following error messages keep appearing and it shows the pod or fluent bit keep sourcing AWS credential from the underlying EKS worker node (EC2 instance) rather than the annotated EKS IAM Roles for Service Account (IRSA).

[2020/10/16 09:52:24] [error] [output:es:es.3] HTTP status=403 URI=/_bulk, response: {"error":{"root_cause":[{"type":"security_exception","reason":"no permissions for [indices:data/write/bulk] and User [name=arn:aws:iam::XXX873347XXX:role/eksctl-cluster-1-nodegroup-ng-al1-NodeInstanceRole-7GZZR0O6HRQS, backend_roles=[arn:aws:iam::XXX873347XXX:role/eksctl-cluster-1-nodegroup-ng-al1-NodeInstanceRole-7GZZR0O6HRQS], requestedTenant=null]"}],"type":"security_exception","reason":"no permissions for [indices:data/write/bulk] and User [name=arn:aws:iam::XXX873347XXX:role/eksctl-cluster-1-nodegroup-ng-al1-NodeInstanceRole-7GZZR0O6HRQS, backend_roles=[arn:aws:iam::XXX873347XXX:role/eksctl-cluster-1-nodegroup-ng-al1-NodeInstanceRole-7GZZR0O6HRQS], requestedTenant=null]"},"status":403}

The config of fluent bit is here:

[OUTPUT]
    Name            es
    Match           kube.*
    Host            amazon-es-domain.ap-southeast-1.es.amazonaws.com
    Port            443
    TLS             On
    Logstash_Format On
    Logstash_Prefix eks-cluster-1
    Retry_Limit     10
    AWS_Auth        On
    AWS_Region      ap-southeast-1
    Generate_ID     On
    Replace_Dots    On

[PettitWesley]

Duplicate of: fluent/fluent-bit#2714

[PettitWesley]

It is a bug; I'm working on fixing it

[PettitWesley]

Downgrade to 2.7.0 to fix it

[hoegertn]

I can confirm that this happens on ECS/Fargate with Firelens also.

Setting AWS_STS_Endpoint as advised in the referenced ticket helps.

[PettitWesley]

This was fixed in Fluent Bit upstream 1.6.2

AWS for Fluent Bit has not released yet because are trying to fix fluent/fluent-bit#2715

[PettitWesley]

This was fixed in the latest release: https://github.com/aws/aws-for-fluent-bit/releases/tag/v2.9.0