Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fluent-bit] Unable to authenticate Opensearch with custom IAM role in EKS 1.24 #383

Open
bogdandisc opened this issue Jun 14, 2023 · 2 comments

Comments

@bogdandisc
Copy link

bogdandisc commented Jun 14, 2023

Environment:

EKS 1.23
fluent-bit chart version 0.30.4
fluent-bit app version 2.0.14

I am trying to deploy fluent-bit using the standard helm chart:

resource "helm_release" "fluent-bit" {
  repository       = "https://fluent.github.io/helm-charts"
  version          = var.versions.fluent_bit_chart
  chart            = "fluent-bit"
  name             = "fluent-bit"
  namespace        = "kube-system"
  create_namespace = false
  atomic           = true

  values = [
    templatefile("${path.module}/templates/fluent-bit-helm-values.yaml", {
      role_arn = "role_arn" 
      ...
    })
  ]
}

Have tried to create a new service account via the helm chart:

serviceAccount:
  create: true
  name: fluent-bit
  annotations:
    eks.amazonaws.com/role-arn: ${role_arn}

And also tried to create a new service account and attach it in the values.yaml file.
In both cases I can see the service account getting attached to the fluent-bit pods and the trust policy working. There's no reason that the IRSA token is not used by the fluent-bit role.

I am also aware that in 1.24 secrets are not created automatically, so I am creating one in terraform and attaching it to the service account:

resource "kubernetes_secret" "fluent_bit" {
  metadata {
    generate_name = "fluent-bit-"
    namespace     = "kube-system"
    annotations = {
      "kubernetes.io/service-account.name" = kubernetes_service_account.fluent_bit_sa.metadata[0].name
    }
  }

  type                           = "kubernetes.io/service-account-token"
  wait_for_service_account_token = true
}

However the fluent-bit pods are not authenticating using the fluent-bit service account and are using the worker node IAM role instead:

23/06/14 09:54:29] [error] [output:opensearch:opensearch-application] HTTP status=403 URI=/_bulk, response:
{"Message":"User: arn:aws:sts::****:assumed-role/eks_role is not authorized to perform: es:ESHttpPost because no identity-based policy allows the es:ESHttpPost action"}
@RobertKarolczuk
Copy link

Hi @bogdandisc have You resolved the issue?
Unfortunately I have similar issue and according to the AWS documentation You have to attach proper policy to the OS, and additionally You have to enable AWS Signature Version 4.

Even if you configure a completely open resource-based access policy, all requests to the OpenSearch Service configuration API must be signed. If your policies specify IAM roles or users, requests to the OpenSearch APIs also must be signed using AWS Signature Version 4.

I found in Fluentbit Docuemtnation the parameter AWS_Auth - Enable AWS Sigv4 Authentication for Amazon OpenSearch Service

but I use helm chart and to be honest I don't know how to use this parameter in my case...
Below section from values.yaml

clusterOutputs:
  - name: opensearch-output
    spec:
      opensearch:
        host: vpc-domain-name-xxxxxxxxxxxxx.us-east-1.es.amazonaws.com
        port: 443
        scheme: https
        logstash_format: true
        logstash_prefix: ${tag}
        index_name: ${tag}-%Y.%m.%d
        include_timestamp: true
        reconnect_on_error: true
        log_os_400_reason: true
        buffer:
          path: /buffers/opensearch
          type: file
          timekey: 1m
          timekey_wait: 30s
          timekey_use_utc: true

Do You have any suggestions/ideas?

Thanks

@dioguerra dioguerra changed the title Unable to authenticate Opensearch with custom IAM role in EKS 1.24 [fluent-bit] Unable to authenticate Opensearch with custom IAM role in EKS 1.24 Aug 31, 2023
@skybalsamoan
Copy link

have you given necessary wite grants on opensearch side? you have to map your role with necessary permission
https://opensearch.org/docs/latest/security/access-control/users-roles/
Creation of k8s secrets was not necessary from my experience
In your output conf you have also to add
AWS_Auth On
AWS_Region eu-west-1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants