Filebeat not picking up all logs

[danielc103]

This may be more of a question than an actual issue.

I've not made any modifications to the filebeat deployment.

Kibana successfully pulls logs for my autoscaler and elk stack. Kibana picks up all my namespaces correctly, but for some reason does not show logs for nginx-ingress, external-dns, or any other services. I've deployed an AWX operator into default namespace, no logs.

Am I missing a setting or flag I should toggle in Kibana or Elasticsearch to pull those?

filebeat:
  enabled: true
  filebeatConfig:
    filebeat.yml: |
      filebeat.modules:
        - module: system
          syslog:
            enabled: true
            #var.paths: ["/var/log/syslog"]
      filebeat.autodiscover:
        providers:
          - type: kubernetes
            node: $${NODE_NAME}
            hints.enabled: true
            hints.default_config:
              type: container
              paths:
                - /var/log/containers/*-$${data.kubernetes.container.id}.log

      processors:
        - drop_event:
            when:
              equals:
                kubernetes.container.name: "filebeat"

        - add_cloud_metadata:
        - add_kubernetes_metadata:
        - add_docker_metadata:

      output.elasticsearch:
        username: '$${ELASTICSEARCH_USERNAME}'
        password: '$${ELASTICSEARCH_PASSWORD}'
        protocol: https
        hosts: ["elasticsearch-master:9200"]
        ssl.verification_mode: none

[danielc103]

I was able to resolve my issue by modifying the input for Filebeat

filebeat:
  enabled: true
  filebeatConfig:
    filebeat.yml: |
      filebeat.inputs:
      - type: container
        paths:
          - /var/log/containers/*.log
        processors:
          - add_kubernetes_metadata:
              host: $${NODE_NAME}
              matchers:
              - logs_path:
                  logs_path: "/var/log/containers/"
      filebeat.modules:
        - module: system
          syslog:
            enabled: true
            #var.paths: ["/var/log/syslog"]
      filebeat.autodiscover:
        providers:
          - type: kubernetes
            node: $${NODE_NAME}
            hints.enabled: true
            hints.default_config:
              type: container
              paths:
                - /var/log/containers/*-$${data.kubernetes.container.id}.log

      processors:
        - drop_event:
            when:
              equals:
                kubernetes.container.name: "filebeat"

        - add_cloud_metadata:
        - add_kubernetes_metadata:
        - add_docker_metadata:

      output.elasticsearch:
        username: '$${ELASTICSEARCH_USERNAME}'
        password: '$${ELASTICSEARCH_PASSWORD}'
        protocol: https
        hosts: ["elasticsearch-master:9200"]
        ssl.verification_mode: none

This config successfully pulls all the container logs into Kibana (Elasticsearch). Not sure if this is what you intended. On that same note, I was wondering what purpose does running Loki and Filebeat together serve?

[halfb00t]

Hi @danielc103,

thank you for your report!

We got rid of - type: container after adding filebeat.autodiscover and everything worked well.
I'll try to reproduce your issue and will come back later.

I was wondering what purpose does running Loki and Filebeat together serve?

Some time ago we started to use Loki after it became more usable and Prometheus Stack as a default solution to collect metrics and logs and draw dashboards based on that data. But ELK is still the main stack to collect logs for many projects. So we left them both.

But yeah, seems like adding Loki as a default logging service was pretty early.

[khalezin]

Hi @danielc103

I reproduce the problem which you describe. And I found that the filebeat read only ingress-controller logs without Nginx access logs. I found miss-configuration for filebeat autodiscovery and fixed it in this PR: #55

filebeat.autodiscover - automatically adds the Kubernetes labels so I removed conflicting options with autodiscovery.
And I changed paths for logs filebeat autodiscovery as recommended in the filebeat documentation.

After the filebeat configuration was updated. All ingress-Nginx logs, external-dns and other application logs became available.

Thank you for your report.