Unable to STS assume Role in fluentbit/S3 plugin

[sabdalla80]

I am am unable to assume a role all of a sudden from my daemonset/EKS application. The fluentbit application is trying to assume a role in a different account so it can write the logs to a bucket there. I am seeing this error recently without knowing what changed to cause this error. I appreciate any feedback on this.
fluentBitChartVersion=0.46.7
fluentBitImageRepo=fluent/fluent-bit
fluentBitImageTag=2.2.0
fluentBitChartName=fluent-bit

My output:

[OUTPUT]
Name s3
Match internal.*
region us-east-2
bucket customer-logs-bucket-us-east-2
external_id someId
role_arn arn:aws:iam::account2:role/roletoassume
sts_endpoint https://sts.us-east-2.amazonaws.com
store_dir /tmp/fluent-bit/s3-kube
retry_limit 10
total_file_size 15M
upload_timeout 15s
store_dir_limit_size 50M
s3_key_format $TAG-$UUID
s3_key_format_tag_delimiters ._
compression gzip

The error from logs with debug on:

[2024/05/26 19:18:24] [ info] [filter:kubernetes:kubernetes.3] connectivity OK
[2024/05/26 19:18:24] [ info] [input:emitter:tag_for_s3] initializing
[2024/05/26 19:18:24] [ info] [input:emitter:tag_for_s3] storage_strategy='memory' (memory only)
[2024/05/26 19:18:24] [ info] [input:emitter:tag_for_kube] initializing
[2024/05/26 19:18:24] [ info] [input:emitter:tag_for_kube] storage_strategy='memory' (memory only)
[2024/05/26 19:18:24] [ info] [fstore] created root path /tmp/fluent-bit/s3-kube/customer-logs-bucket-us-east-2
[2024/05/26 19:18:24] [ info] [output:s3:s3.0] Using upload size 15000000 bytes
[2024/05/26 19:18:24] [ info] [aws_client] auth error, refreshing creds
[2024/05/26 19:18:24] [error] [aws_credentials] Shared credentials file /root/.aws/credentials does not exist
[2024/05/26 19:18:24] [ info] [output:s3:s3.0] worker #0 started
[2024/05/26 19:18:24] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020

[2024/05/26 19:19:42] [ info] [aws_client] auth error, refreshing creds
[2024/05/26 19:19:42] [error] [aws_credentials] Shared credentials file /root/.aws/credentials does not exist
[2024/05/26 19:19:42] [error] [aws_credentials] STS assume role request failed
[2024/05/26 19:19:42] [ warn] [aws_credentials] No cached credentials are available and a credential refresh is already in progress. The current co-routine will retry.
[2024/05/26 19:19:42] [error] [signv4] Provider returned no credentials, service=s3
[2024/05/26 19:19:42] [error] [aws_client] could not sign request
[2024/05/26 19:19:42] [error] [aws_credentials] STS assume role request failed

[sabdalla80]

@PettitWesley I am seeing the same issue as this one (Fluent Bit 1.6 - ES Plugin: Failed to source credential on Amazon EKS IAM Roles for Service Account #2714). Could the bug have been re-introduced?
I am able to send to S3, but not able to assume the role.

Here is another snippet of debug outputs

[2024/05/26 20:19:48] [debug] [upstream] KA connection #77 to s3.us-east-2.amazonaws.com:443 has been assigned (recycled)
[2024/05/26 20:19:48] [debug] [http_client] not using http_proxy for header
[2024/05/26 20:19:48] [debug] [aws_credentials] Requesting credentials from the STS provider..
[2024/05/26 20:19:48] [debug] [aws_credentials] STS Provider: Refreshing credential cache.
[2024/05/26 20:19:48] [debug] [aws_credentials] Calling STS..
[2024/05/26 20:19:48] [debug] [upstream] KA connection #343 to sts.us-east-2.amazonaws.com:443 has been assigned (recycled)
[2024/05/26 20:19:48] [debug] [http_client] not using http_proxy for header
[2024/05/26 20:19:48] [debug] [aws_credentials] Requesting credentials from the EKS provider..
[2024/05/26 20:19:48] [debug] [task] destroy task=0x7fd750366f00 (task_id=0)
[2024/05/26 20:19:48] [debug] [task] created task=0x7fd750366f00 id=0 without routes, dropping.
[2024/05/26 20:19:48] [debug] [task] destroy task=0x7fd750366f00 (task_id=0)
[2024/05/26 20:19:48] [debug] [task] created task=0x7fd750366f00 id=0 without routes, dropping.

[2024/05/26 20:19:48] [debug] [task] destroy task=0x7fd750366f00 (task_id=0)
[2024/05/26 20:19:48] [debug] [aws_client] sts.us-east-2.amazonaws.com: http_do=0, HTTP Status: 403
[2024/05/26 20:19:48] [debug] [upstream] KA connection #343 to sts.us-east-2.amazonaws.com:443 is now available
[2024/05/26 20:19:48] [debug] [aws_client] Unable to parse API response- response is not valid JSON.
[2024/05/26 20:19:48] [debug] [aws_credentials] STS raw response:

[PettitWesley]

Use latest or latest stable version