Could not communicate to OpenSearch ... No such file or directory @ rb_sysopen - (Errno::ENOENT)

[yoav-klein]

(check apply)

• [ x] read the contribution guideline
• (optional) already reported 3rd party upstream repository or mailing list if you use k8s addon or helm charts.

Steps to replicate

• Create a EKS cluster in AWS
• Create a OpenSearch domain in the same VPC
• Run the following configuration in the cluster:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: kube-system
  labels:
    k8s-app: fluentd-logging
    version: v1
spec:
  selector:
    matchLabels:
      k8s-app: fluentd-logging
      version: v1
  template:
    metadata:
      labels:
        k8s-app: fluentd-logging
        version: v1
    spec:
      serviceAccountName: fluentd-es
      tolerations:
      - key: node-role.kubernetes.io/control-plane
        effect: NoSchedule
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      containers:
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:v1-debian-opensearch
        env:
          - name: K8S_NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name: FLUENT_OPENSEARCH_HOST
            #value: opensearch.storage.svc.cluster.local
            value: xyz.us-east-1.es.amazonaws.com
          - name: FLUENT_OPENSEARCH_PORT
            value: "443"
          - name: FLUENT_OPENSEARCH_INDEX_NAME
            value: fluentd
          - name: FLUENT_OPENSEARCH_SSL_VERIFY
            value: "false"
          - name: FLUENT_OPENSEARCH_SCHEME
            value: "https"
          - name: FLUENT_OPENSEARCH_LOGSTASH_FORMAT
            value: "true"
          - name: FLUENT_CONTAINER_TAIL_PARSER_TYPE
            value: "cri"
          - name: FLUENT_CONTAINER_TAIL_PARSER_TIME_FORMAT
            value: "%Y-%m-%dT%H:%M:%S.%N%:z"
         resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: dockercontainerlogdirectory
          mountPath: /var/log/pods
          readOnly: true
        - name: config
          mountPath: /fluentd/etc/conf.d
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      # When actual pod logs in /var/lib/docker/containers, the following lines should be used.
      # - name: dockercontainerlogdirectory
      #   hostPath:
      #     path: /var/lib/docker/containers
      # When actual pod logs in /var/log/pods, the following lines should be used.
      - name: dockercontainerlogdirectory
        hostPath:
          path: /var/log/pods
      - name: config
        configMap:
          name: fluentd

Expected Behavior or What you need to ask

I expect that fluentd will send logs to OpenSearch, but instead I get the following error:

$ kubectl logs -n kube-system fluentd-g4fe3
...
2023-04-23 17:36:16 +0000 [warn]: #0 [out_os] Remaining retry: 8. Retry to communicate after 128 second(s).
2023-04-23 17:40:32 +0000 [warn]: #0 [out_os] Could not communicate to OpenSearch, resetting connection and trying again. No such file or directory @ rb_sysopen -  (Errno::ENOENT)
...

...

Using Fluentd and OpenSearch plugin versions

fluent-plugin-opensearch 1.0.10
fluentd 1.15.3

• OS version
  Running in a debian based Docker image
• Bare Metal or within Docker or Kubernetes or others?
  Runnin in AWS EKS
• Fluentd v1.0 or later
  • paste result of fluentd --version or td-agent --version
• OpenSearch plugin version
  • paste boot log of fluentd or td-agent
  • paste result of fluent-gem list, td-agent-gem list or your Gemfile.lock
• OpenSearch version (optional)
  AWS OpenSearch 2.3
• OpenSearch template(s) (optional)

[Jared0430]

I'm getting the same issue

[Jared0430]

Ok I just found a fix;

You need to create a ConfigMap to override the fluent.conf file and mount it to the DaemonSet

Config contents taken from https://github.com/fluent/fluentd-kubernetes-daemonset/blob/master/docker-image/v1.16/debian-opensearch/conf/fluent.conf, I've just removed the following lines, as we don't need to mount these files when using cloud OpenSearch in AWS

  ca_file "#{ENV['FLUENT_OPENSEARCH_CA_PATH']}"
  client_cert "#{ENV['FLUENT_OPENSEARCH_CLIENT_CERT']}"
  client_key "#{ENV['FLUENT_OPENSEARCH_CLIENT_KEY']}"
  client_key_pass "#{ENV['FLUENT_OPENSEARCH_CLIENT_KEY_PASS']}"

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd
  namespace: logging
data:
  fluent.conf: |
    @include "#{ENV['FLUENTD_SYSTEMD_CONF'] || 'systemd'}.conf"
    @include "#{ENV['FLUENTD_PROMETHEUS_CONF'] || 'prometheus'}.conf"
    @include kubernetes.conf
    @include conf.d/*.conf

    <match **>
      @type opensearch
      @id out_os
      @log_level info
      include_tag_key true
      host "#{ENV['FLUENT_OPENSEARCH_HOST']}"
      port "#{ENV['FLUENT_OPENSEARCH_PORT']}"
      path "#{ENV['FLUENT_OPENSEARCH_PATH']}"
      scheme "#{ENV['FLUENT_OPENSEARCH_SCHEME'] || 'http'}"
      ssl_verify "#{ENV['FLUENT_OPENSEARCH_SSL_VERIFY'] || 'true'}"
      ssl_version "#{ENV['FLUENT_OPENSEARCH_SSL_VERSION'] || 'TLSv1_2'}"
      user "#{ENV['FLUENT_OPENSEARCH_USER']}"
      password "#{ENV['FLUENT_OPENSEARCH_PASSWORD']}"
      index_name "#{ENV['FLUENT_OPENSEARCH_INDEX_NAME'] || 'fluentd'}"
      logstash_dateformat "#{ENV['FLUENT_OPENSEARCH_LOGSTASH_DATEFORMAT'] || '%Y.%m.%d'}"
      logstash_format "#{ENV['FLUENT_OPENSEARCH_LOGSTASH_FORMAT'] || 'false'}"
      logstash_prefix "#{ENV['FLUENT_OPENSEARCH_LOGSTASH_PREFIX'] || 'logstash'}"
      logstash_prefix_separator "#{ENV['FLUENT_OPENSEARCH_LOGSTASH_PREFIX_SEPARATOR'] || '-'}"
      <buffer>
        flush_thread_count "#{ENV['FLUENT_OPENSEARCH_BUFFER_FLUSH_THREAD_COUNT'] || '1'}"
        flush_mode "#{ENV['FLUENT_OPENSEARCH_BUFFER_FLUSH_MODE'] || 'interval'}"
        flush_interval "#{ENV['FLUENT_OPENSEARCH_BUFFER_FLUSH_INTERVAL'] || '60s'}"
        chunk_limit_size "#{ENV['FLUENT_OPENSEARCH_BUFFER_CHUNK_LIMIT_SIZE'] || '8M'}"
        total_limit_size "#{ENV['FLUENT_OPENSEARCH_BUFFER_TOTAL_LIMIT_SIZE'] || '512M'}"
        retry_max_interval "#{ENV['FLUENT_OPENSEARCH_BUFFER_RETRY_MAX_INTERVAL'] || '30'}"
        retry_timeout "#{ENV['FLUENT_OPENSEARCH_BUFFER_RETRY_TIMEOUT'] || '72h'}"
        retry_forever "#{ENV['FLUENT_OPENSEARCH_BUFFER_RETRY_FOREVER'] || 'false'}"
      </buffer>
    </match>

You then need to mount the ConfigMap like this

apiVersion: apps/v1
kind: DaemonSet
spec:
  template:
    spec:
      containers:
        - name: fluentd-logging
          # ...
          volumeMounts:
            # ...
            - name: fluent-config
              mountPath: /fluentd/etc/fluent.conf
              subPath: fluent.conf
      volumes:
        # ...
        - name: fluent-config
          configMap:
            name: fluentd

[yoav-klein]

@Jared0430 , thanks ! found the same fix also :)

[cosmo0920]

Seems to be solved. Closing.