Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update ECK-Beats default values to not include ElasticsearchRef. #7228

Merged
merged 1 commit into from Oct 12, 2023

Conversation

naemono
Copy link
Contributor

@naemono naemono commented Oct 12, 2023

see helm/helm#12469 (comment)
related #7143

The ongoing Helm parent->child default values override issue continues.

If you currently attempt to use eck-stack with eck-logstash in the above PR with the following example (based off of logstash-eck.yaml recipe ), it will fail as you cannot null out the default elasticsearchRef in the eck-beats helm chart:

---
eck-elasticsearch:
  version: 8.9.0
  nodeSets:
  - name: default
    count: 3
    config:
      node.store.allow_mmap: false
    podTemplate:
      spec:
        containers:
        - name: elasticsearch
          resources:
            limits:
              memory: 2Gi
            requests:
              memory: 2Gi
eck-beats:
  enabled: true
  version: 8.9.0
  spec:
    type: filebeat
    daemonSet: null
    elasticsearchRef: null
    config:
      filebeat.inputs:
      - type: log
        paths:
          - /data/logstash-tutorial.log
      processors:
      - add_host_metadata: {}
      - add_cloud_metadata: {}
      output.logstash:
        hosts: ["logstash-ls-beats:5044"]
    deployment:
      podTemplate:
        spec:
          automountServiceAccountToken: true
          initContainers:
            - name: download-tutorial
              image: curlimages/curl
              command: ["/bin/sh"]
              args: ["-c", "curl -L https://download.elastic.co/demos/logstash/gettingstarted/logstash-tutorial.log.gz | gunzip -c > /data/logstash-tutorial.log"]
              volumeMounts:
                - name: data
                  mountPath: /data
          containers:
            - name: filebeat
              securityContext:
                runAsUser: 1000
              volumeMounts:
                - name: data
                  mountPath: /data
                - name: beat-data
                  mountPath: /usr/share/filebeat/data
          volumes:
            - name: data
              emptydir: {}
            - name: beat-data
              emptydir: {}
eck-logstash:
  enabled: true
  version: 8.9.0
  elasticsearchRefs:
    - clusterName: eck
      name: elasticsearch
  pipelines:
    - pipeline.id: main
      config.string: |
        input {
          beats {
            port => 5044
          }
        }
        filter {
          grok {
            match => { "message" => "%{HTTPD_COMMONLOG}"}
          }
          geoip {
            source => "[source][address]"
            target => "[source]"
          }
        }
        output {
          elasticsearch {
            hosts => [ "${ECK_ES_HOSTS}" ]
            user => "${ECK_ES_USER}"
            password => "${ECK_ES_PASSWORD}"
            ssl_certificate_authorities => "${ECK_ES_SSL_CERTIFICATE_AUTHORITY}"
          }
        }

  services:
    - name: beats
      service:
        spec:
          type: ClusterIP
          ports:
            - port: 5044
              name: "filebeat"
              protocol: TCP
              targetPort: 5044

It will configure beat as such:

❯ kubedecode logstash-stack-eck-beats-beat-filebeat-config elastic
beat.yml: filebeat:
    inputs:
        - paths:
            - /data/logstash-tutorial.log
          type: log
output:
    elasticsearch:
        hosts:
            - https://elasticsearch-es-http.elastic.svc:9200
        password: x4R7M2812tns497DgFdPfTQ6
        ssl:
            certificate_authorities:
                - /mnt/elastic-internal/elasticsearch-certs/ca.crt
        username: elastic-logstash-stack-eck-beats-beat-user
    logstash:
        hosts:
            - logstash-ls-beats:5044
processors:
    - add_host_metadata: null
    - add_cloud_metadata: null

Which isn't valid as you can't have 2x outputs in beat.

With this change you will get this, which is valid:

kubedecode logstash-stack-eck-beats-beat-filebeat-config elastic
beat.yml: filebeat:
    inputs:
        - paths:
            - /data/logstash-tutorial.log
          type: log
output:
    logstash:
        hosts:
            - logstash-ls-beats:5044
processors:
    - add_host_metadata: null
    - add_cloud_metadata: null

I'm suggesting we remove the default elasticsearchRef we include in the eck-beast chart for now until this issue is resolved.

Signed-off-by: Michael Montgomery <mmontg1@gmail.com>
@naemono naemono added the >bug Something isn't working label Oct 12, 2023
Copy link
Contributor

@thbkrkr thbkrkr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like a good compromise given the current behavior/bug.

@naemono naemono merged commit 32b3f2e into elastic:main Oct 12, 2023
5 checks passed
@naemono naemono deleted the remove-default-values-eck-beats branch October 12, 2023 13:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>bug Something isn't working v2.10.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants