Fluent Bit 1.6 - ES Plugin: Failed to source credential on Amazon EKS IAM Roles for Service Account #2714
Comments
|
seeing the exact same problem, I installed using the helm chart from google stable repo, here's the manifests that end up in the cluster: apiVersion: v1
kind: ServiceAccount
metadata:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::my-acc:role/fluent-bit
meta.helm.sh/release-name: fluent-bit
meta.helm.sh/release-namespace: logging
labels:
app: fluent-bit
app.kubernetes.io/managed-by: Helm
chart: fluent-bit-2.10.1
heritage: Helm
release: fluent-bit
name: fluent-bit
namespace: loggingapiVersion: v1
kind: Pod
metadata:
annotations:
kubernetes.io/psp: eks.privileged
labels:
app: fluent-bit
controller-revision-hash: 7d55f48cd8
pod-template-generation: "1"
release: fluent-bit
name: fluent-bit-2g698
namespace: logging
spec:
containers:
- env:
- name: AWS_DEFAULT_REGION
value: eu-west-1
- name: HOSTNAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: AWS_ROLE_ARN
value: arn:aws:iam::my-acc:role/fluent-bit
- name: AWS_WEB_IDENTITY_TOKEN_FILE
value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
image: fluent/fluent-bit:1.6-debug
imagePullPolicy: Always
name: fluent-bit
resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 100m
memory: 128Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/log
name: varlog
- mountPath: /var/lib/docker/containers
name: varlibdockercontainers
readOnly: true
- mountPath: /fluent-bit/etc/fluent-bit.conf
name: config
subPath: fluent-bit.conf
- mountPath: /fluent-bit/etc/fluent-bit-service.conf
name: config
subPath: fluent-bit-service.conf
- mountPath: /fluent-bit/etc/fluent-bit-input.conf
name: config
subPath: fluent-bit-input.conf
- mountPath: /fluent-bit/etc/fluent-bit-filter.conf
name: config
subPath: fluent-bit-filter.conf
- mountPath: /fluent-bit/etc/fluent-bit-output.conf
name: config
subPath: fluent-bit-output.conf
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: fluent-bit-token-65zvs
readOnly: true
- mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
name: aws-iam-token
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: fluent-bit
serviceAccountName: fluent-bit
volumes:
- name: aws-iam-token
projected:
defaultMode: 420
sources:
- serviceAccountToken:
audience: sts.amazonaws.com
expirationSeconds: 86400
path: token
- hostPath:
path: /var/log
type: ""
name: varlog
- hostPath:
path: /var/lib/docker/containers
type: ""
name: varlibdockercontainers
- configMap:
defaultMode: 420
name: fluent-bit-config
name: config
- name: fluent-bit-token-65zvs
secret:
defaultMode: 420
secretName: fluent-bit-token-65zvsand the fluent-bit config: and the fluent-bit logs: The role mentioned in that last log statement is the instance / node profile, the same issue described by the OP. |
|
I have progressed the issue by following this advice to block access to the node role, now the fluent-bit logs read: |
|
A noble stranger on the provider-aws kubernetes slack channel gave me a workaround that fixes this issue for myself and the stranger, specify the AWS_STS_Endpoint in the OUTPUT config: |
|
Hi there! I am the 'noble stranger' mentioned above. 😅 Apologies for not filing the bug beforehand, I thought it was just something weird with the AWS account I was using. Anyhow, I see that no one has posted debug logs for this yet, so I'll post this snippet from mine from when I ran into this issue last week, since that's what led me to go down the STS endpoint config path: It may also be worth noting that I am using the |
PettitWesley
added
the
AWS
|
I think this is probably a bug... IAM Roles for SA calls STS... we made a change to the STS endpoint code to enable custom endpoints. I bet there's a bug there... |
|
I can confirm that this happens on ECS/Fargate with Firelens also. Setting AWS_STS_Endpoint helps. |
|
@hoegertn Are you specifying an IAM role with the aws_role_arn parameter? I'm about to put up a PR to fix this... basically calling STS is broken (which happens if you use EKS IRSA or a custom role). |
|
Yes, I am assuming a role that has ES permissions. As you mentioned the STS call is broken as it does not know the hostname to contact. |
|
Yeah, basically it's because the config map sets https://github.com/fluent/fluent-bit/blob/master/plugins/out_es/es.c#L804 At least that's what I'm testing right now.. |
Signed-off-by: Wesley Pettit <wppttt@amazon.com>
Signed-off-by: Wesley Pettit <wppttt@amazon.com>
|
This was fixed in 1.6.2 AWS for Fluent Bit has not been updated yet since we are still trying to fix #2715 |
|
Is this really fixed? |
|
@ypicard Yes. Please open a new issue if you are having credential issues: https://github.com/aws/aws-for-fluent-bit |
|
2 years have passed, and the issue still exists.
|
|
@tejarora is also happening in somehow for me, getting |
Bug Report
Describe the bug
Fluent Bit 1.6 - ES Plugin: Keep sourcing credential from EC2 instance rather than IAM Roles for Service Account on Amazon EKS Worker Node
To Reproduce
The text was updated successfully, but these errors were encountered: