[BUG] Logstash shows error: Failed to install template

[dk-do]

Describe the bug
While logstash is starting (image: opensearchproject/logstash-oss-with-opensearch-output-plugin:7.16.2) the following messages are shown:

opensearch-logstash-0 logstash [2022-08-12T08:38:41,264][ERROR][logstash.outputs.opensearch][main] Failed to install template 
{:message=>"Failed to load default template for OpenSearch v2 with ECS disabled; caused by: 
#<ArgumentError: 
Template file '/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/outputs/opensearch/templates/ecs-disabled/2x.json' could not be found>", 
:exception=>RuntimeError, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/outputs/opensearch/template_manager.rb:33:in `load_default_template'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/outputs/opensearch/template_manager.rb:21:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/outputs/opensearch.rb:412:in `insta
ll_template'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/outputs/opensearch.rb:247:in `finish_register'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/outputs/opensearch.rb:224:in `block in register'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/plugin_mixins/opensearch/common.rb:83:in `block in after_successful_connection'"]}            

To Reproduce
Steps to reproduce the behavior:
I have installed OpenSearch by using Helm in our local Kubernetes Cluster (1.24.1) and create a statefulset definition for logstash.
This is the statefulset manifest:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  annotations:
  labels:
    app: opensearch-logstash
  name: opensearch-logstash
  namespace: opensearch
spec:
  podManagementPolicy: Parallel
  replicas: 2
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: opensearch-logstash
  serviceName: opensearch-logstash-headless
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: opensearch-logstash
      name: opensearch-logstash
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - opensearch-logstash
            topologyKey: kubernetes.io/hostname
      containers:
      - env:
        - name: LS_JAVA_OPTS
          value: -Xmx1g -Xms1g
        image: opensearchproject/logstash-oss-with-opensearch-output-plugin:7.16.2 
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /
            port: http
            scheme: HTTP
          initialDelaySeconds: 300
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        name: logstash
        ports:
        - containerPort: 9600
          name: http
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /
            port: http
            scheme: HTTP
          initialDelaySeconds: 60
          periodSeconds: 10
          successThreshold: 3
          timeoutSeconds: 5
        resources:
          limits:
            cpu: "1"
            memory: 1536Mi
          requests:
            cpu: 100m
            memory: 1536Mi
        securityContext:
          capabilities:
            drop:
            - ALL
          runAsNonRoot: true
          runAsUser: 1000
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /usr/share/logstash/data
          name: opensearch-logstash
        - mountPath: /usr/share/logstash/pipeline/opensearch-pipeline.conf
          name: logstashpipeline
          subPath: opensearch-pipeline.conf
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 1000
        runAsUser: 1000
      terminationGracePeriodSeconds: 120
      volumes:
      - configMap:
          defaultMode: 420
          name: opensearch-logstash-pipeline 
        name: logstashpipeline
  updateStrategy:
    type: RollingUpdate
  volumeClaimTemplates:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      creationTimestamp: null
      name: opensearch-logstash
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 5Gi
      storageClassName: rook-ceph-block
      volumeMode: Filesystem

In addition I created a new pipeline definition as ConfigMap. It uses Kafka as the input channel:

apiVersion: v1
data:
  opensearch-pipeline.conf: |
    input {
      kafka {
      bootstrap_servers => "kafka-cluster-kafka-bootstrap.kafka.svc.cluster.local:9092"
      codec => json { charset => "UTF-8" }
      group_id => "opensearch-logstash"
      consumer_threads => 10
      sasl_mechanism => 'SCRAM-SHA-512'
      security_protocol => 'SASL_PLAINTEXT'
      sasl_jaas_config => "org.apache.kafka.common.security.scram.ScramLoginModule required username='kafka' password='passw0rd';"

      }
    }
    output {
      if [level] == "DEBUG" {
        opensearch {
          hosts => "http://opensearch-cluster-master:9200"
          manage_template => true
          index => "logstash-debug-%{+YYYY.MM.dd}"
          #document_type => "%{[@metadata][type]}"
        }
      }
      elseif [level] == "VERBOSE" {
      opensearch {
        hosts => "http://opensearch-cluster-master:9200"
        manage_template => true
        index => "logstash-verbose-%{+YYYY.MM.dd}"
      }
      }
      else {
        opensearch {
          hosts => "http://opensearch-cluster-master:9200"
          manage_template => true
          index => "logstash-%{+YYYY.MM.dd}"
          #document_type => "%{[@metadata][type]}"
        }
      }
    }
kind: ConfigMap
metadata:
  labels:
    app: opensearch-logstash
  name: opensearch-logstash-pipeline
  namespace: opensearch

For evaluation I disabled the security plugin.

Expected behavior
Startup without errors or explanation why these error occur.

Plugins
In OpenSearch the "ingest-attachment" plugin.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: Ubuntu
• Version 20.04. LTS

Additional context
None

[dk-do]

Any news on this?

[dblock]

I'll move it to the devops repo, let's see if we can get some attention there.

[prudhvigodithi]

Hey @dk-do can you enable the debug mode for logstash and paste the error log please?
https://opensearch.org/docs/latest/clients/logstash/index/

Also found a smilier issue, looks like this was fixed but not in new release.
But debug mode should help us give some information.

Thank you

[dk-do]

ls_logs.txt

Please find enclosed the debug log.
The first occurence of the error message is in line 634

[prudhvigodithi]

Hey @dk-do I see this was fixed for latest 1.3.0 release https://github.com/opensearch-project/logstash-output-opensearch/releases/tag/v1.3.0 and docker image is yet to be released.
Thank you
@dlvenable @bbarani

[bbarani]

@dk-do , We recently released version 8.4.0 on Docker. Please download and let us know if your issue is fixed in the latest version.

[prudhvigodithi]

Hey @dk-do just checking back to see if you have tried with the latest logstash release and you still see this issue?

[dk-do]

Hey @prudhvigodithi @bbarani,
sorry, for the late reply - I was on a business travel.

Everythings fine now! Thx for your support!