AWS IAM authentication doesn't work properly with IRSA on Kubernetes

[ialejandro]

Based on this comment #275 (comment)

We're using Redpanda Console as a k8s deployment on EKS deployed using the official Helm chart, with IRSA authentication. Everything works well after initial deployment, but after exactly 1 hour, we start getting SASL_AUTHENTICATION_FAILED errors.

By default, the IRSA token has duration of 1h and I guess the problem is that there is no token refresh implemented. Looking at the code here, it looks like that is indeed the case - token is fetched at startup and reused for all subsequent requests. That works until the token expires, after which all requests fail as unauthenticated.

Here is the config we use for Redpanda Console and for the service account:

console:
  config:
    kafka:
      brokers:
        - b-1.xxx.yyyyyy.zz.kafka.eu-central-1.amazonaws.com:9098
        - b-2.xxx.yyyyyy.zz.kafka.eu-central-1.amazonaws.com:9098
        - b-3.xxx.yyyyyy.zz.kafka.eu-central-1.amazonaws.com:9098
      sasl:
        enabled: true
        mechanism: AWS_MSK_IAM
      tls:
        enabled: true

Here is the config we use for service account:

serviceAccount:
  create: true
  automountServiceAccountToken: true
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789:role/something

And here is the error log after one hour, once the token expires:

{"level":"error","ts":"2024-05-02T08:43:12.242Z","logger":"kafka_client","msg":"unable to initialize sasl","broker":"2","err":"[55396e20-7c33-4065-b9bd-fb7413070759]: Access denied: SASL_AUTHENTICATION_FAILED: SASL Authentication failed."}

Kafka Console release

• Helm chart: console-0.7.26
• Redpanda console version: v2.4.6

Mention @mislavmandaricaxilis

[mislavmandaricaxilis]

Hey 👋

Here's a PR addressing this issue.