assume role via role_arn for s3 output doesn't seem to work

[mfractal]

we’re running on top of aws eks, and we have hashicorp vault cluster in there. i am trying to configure fluentbit to upload vault audit logs to s3. everything works fine if i provide AWS_ACCESS_KEY_ID and SECRET via env variables but we would like to use the role_arn option utilizing IRSA.
I think i setup the role/policies/trust relationship correctly but i still getting errors.
See logs.

Configuration

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
data:
  fluent-bit.conf: |
    [INPUT]
        Name              tail
        Tag               vault.audit
        Path              /vault/audit/*.txt
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  10
        Ignore_Older      1h

    [OUTPUT]
        Name                  s3
        Match                 *
        bucket                avalor-vault-audit-stg
        region                us-east-1
        s3_key_format         /vault/$TAG/%Y/%m/%d/%H/%M/%S
        total_file_size       1M
        upload_timeout        1m
        use_put_object        On
        role_arn              arn:aws:iam::XXX:role/FluentBitS3Role
        sts_endpoint          https://sts.eu-west-1.amazonaws.com 

IRSA policy

{
    "Statement": [
        {
            "Action": [
                "kms:Decrypt",
                "kms:Encrypt",
                "kms:DescribeKey"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:kms:us-west-2:ACCOUNT_NUMBER:key/mrk-KEY_ID",
            "Sid": "KMS"
        },
        {
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:PutObjectAcl",
                "s3:DeleteObject",
                "s3:ListBucket"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::TEST_BUCKET_NAME/*",
                "arn:aws:s3:::TEST_BUCKET_NAME"
            ],
            "Sid": "S3"
        },
        {
            "Action": [
                "iam:AttachUserPolicy",
                "iam:CreateAccessKey",
                "iam:CreateUser",
                "iam:DeleteAccessKey",
                "iam:DeleteUser",
                "iam:DeleteUserPolicy",
                "iam:DetachUserPolicy",
                "iam:ListAccessKeys",
                "iam:ListAttachedUserPolicies",
                "iam:ListGroupsForUser",
                "iam:ListUserPolicies",
                "iam:PutUserPolicy",
                "iam:RemoveUserFromGroup",
                "sts:AssumeRole",
                "ec2:DescribeInstances",
                "iam:GetInstanceProfile",
                "iam:GetUser",
                "iam:GetRole"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ],
    "Version": "2012-10-17"
}

AWS IRSA Trust Relationship:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::ACCOUNT_NUMBER:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/OIDC_NUMBER"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.us-west-2.amazonaws.com/id/OIDC_NUMBER": "sts.amazonaws.com",
                    "oidc.eks.us-west-2.amazonaws.com/id/OIDC_NUMBER": "system:serviceaccount:vault:vault"
                }
            }
        }
    ]
}

Fluent Bit Log Output

│ fluent-bit [2023/03/06 19:15:23] [ warn] [aws_credentials] No cached credentials are available and a credential refresh is already in progress. The current co-routine will retry.                                                                                                                                  │
│ fluent-bit [2023/03/06 19:15:23] [error] [signv4] Provider returned no credentials, service=s3                                                                                                                                                                                                                      │
│ fluent-bit [2023/03/06 19:15:23] [error] [aws_client] could not sign request                                                                                                                                                                                                                                        │
│ fluent-bit [2023/03/06 19:15:23] [error] [aws_credentials] STS assume role request failed                                                                                                                                                                                                                                                                       

Fluent Bit Version Info

public.ecr.aws/aws-observability/aws-for-fluent-bit:arm64-2.31.5

Related Issues

This and this might be related ?

[mfractal]

i should add that i just tested vector.dev and it works fine with assume role so aws configuration is done correctly.

[PettitWesley]

It looks like it can't find the base IRSA credentials. Enable debug logging to get more info: https://github.com/aws/aws-for-fluent-bit/blob/mainline/troubleshooting/debugging.md#enable-debug-logging

You need base creds to then assume a role. I'd try to get IRSA working first to put to S3, then add the extra assume role step.

[ad-m-ss]

Could you get into the debug container and execute aws sts get-caller-identity to make sure that IRSA is working fine?
Could you share the manifest of your Kubernetes service account used for Fluentbit?

In my opinion, the problem is that you are configuring the IRSA incorrectly. IRSA should be configured as an annotation to Kubernetes service-account, instead of being indicated as a role inside a container that cannot authenticate to AWS.

I just configured AWS EKS + AWS IRSA to send logs using Fluentbit to AWS CloudWatch.

[PettitWesley]

We have debug images here: https://github.com/aws/aws-for-fluent-bit#Using-the-debug-images