Filebeat NGINX module 7.10.0 Upgrade Errors

[christophercutajar]

For confirmed bugs, please report:

• Version: 7.10.0
• Operating System: ECK 1.3
• Steps to Reproduce: Upgrade filebeat with nginx module from 7.9.3 to 7.10.0

While upgrading filebeat from 7.9.3 to 7.10.0 which leverage a nginx module the deployment was failing. After checking the log file I had the below errors:

2020-11-12T11:04:12.865Z        ERROR   instance/beat.go:951    Exiting: 1 error: Error setting up ML for nginx_ecs: 10 errors: ; ; ; ; ; ; ; ; ; 
Exiting: 1 error: Error setting up ML for nginx_ecs: 10 errors: ; ; ; ; ; ; ; ; ;

2020-11-12T11:37:14.792Z        ERROR   instance/beat.go:951    Exiting: 1 error: Error setting up ML for nginx_ecs: cannot set up ML with prefix: filebeat-nginx_ecs-access-, response: 
Exiting: 1 error: Error setting up ML for nginx_ecs: cannot set up ML with prefix: filebeat-nginx_ecs-access-, response:

[elasticmachine]

Pinging @elastic/integrations-services (Team:Services)

[christophercutajar]

filebeat.yml looks like this:

setup.dashboards.enabled: true
setup.template.enabled: true
setup.template.overwrite: true
setup.kibana:
host: "xxxxxx-kb-http:5601"
username: ${ELASTIC_USERNAME}
password: ${ELASTIC_PASSWORD}
protocol: https
ssl:
  enabled: true
  certificate_authorities: /usr/share/filebeat/pki/kibana.crt
  supported_protocols: ["TLSv1.2"]
  renegotiation: never

filebeat.modules:
- module: elasticsearch
- module: logstash
- module: kibana
- module: nginx

output.elasticsearch:
  hosts: ["xxxxxx-es-http:9200"]
  loadbalance: true
  max_retries: 0 # retry forever
  username: ${ELASTIC_USERNAME}
  password: ${ELASTIC_PASSWORD}
  protocol: https
  ssl:
    enabled: true
    certificate_authorities: /usr/share/filebeat/pki/elasticsearch.crt
    supported_protocols: ["TLSv1.2"]
    renegotiation: never

[christophercutajar]

Linking some issues provided by @sophiec20 in relation to this issue:

• https://github.com/elastic/sdh-beats/issues/460
• Filebeat should not setup ML modules if the index pattern does not exist #11349

Using the elastic user still keeps getting the same error logs.

[christophercutajar]

@kvch By adding --dashboards --index-management to the filebeat -e setup in our Helm configuration and leaving the modules configured in the yml file, did the job!

What now I'm not sure is how to load the ML jobs for nginx, unless they are pre-loaded!

[EvgeniGordeev]

@christophercutajar filebeat setup -e --modules nginx --dashboards --index-management didn't help in our case (Kubernetes 1.16 cluster, ingress-nginx v0.40.2), actually also tried to upgrade to 7.10.1 but without luck. While checking events on the Discover tab I don't see any hits with event.module:nginx as they used to be in 7.9.3.

BTW the dashboards were recreated in Kibana but now [Filebeat Nginx] Overview ECS gives errors like Saved field "source.geo.location" is invalid for use with the "Geohash" aggregation. Please select a new field. and Saved field "user_agent.version" is invalid for use with the "Terms" aggregation. Please select a new field.. Which leads to an assumption that dashboard piece inside filebeat module directory is not compatible with the latest elasticsearch version.

filebeat.yml:

      filebeat.inputs:
      - type: container
        paths:
          - /var/log/containers/*.log
        processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"
      filebeat.config.modules:
        path: ${path.config}/modules.d/*.yml
      fields:
        logtype: kubernetes
        kubernetes.cluster.name: '${EKS_CLUSTER_NAME:undefined}'
        k8s.cluster.name: '${EKS_CLUSTER_NAME:undefined}'
      setup.kibana:
        host: "elastic-stack-kibana:5601"
        protocol: "http"
      output.elasticsearch:
        host: '${NODE_NAME}'
        hosts: '${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}'
      # setup custom index name since we use a single ES for multiple clusters
      setup.ilm:
        rollover_alias: 'filebeat-${EKS_CLUSTER_NAME:undefined}-%{[agent.version]}'
        overwrite: true
      setup.template.settings:
        index:
          max_docvalue_fields_search: 200
      filebeat.autodiscover:
        providers:
          - type: kubernetes
            templates:
              - condition:
                  equals:
                    kubernetes.labels.app: ingress-nginx
                config:
                  - module: nginx
                    error:
                      enabled: true
                    ingress_controller:
                      enabled: true
                      input:
                        type: container
                        containers.ids:
                          - "${data.kubernetes.container.id}"

[EvgeniGordeev]

still the same issue with 7.10.2 - loading dashboards with filebeat (filebeat setup -e --modules nginx --dashboards --index-management) did NOT help

[roysG]

Same for me, any update?

[christophercutajar]

Same for me, any update?

What version are you running @roysG?

[roysG]

Version of filebeat?

…

On Tue, 20 Jul 2021, 15:16 Christopher Cutajar ***@***.***> wrote: Same for me, any update? What version are you running @roysG <https://github.com/roysG>? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#22567 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABXZDOVHWE2BLIB47TOWBWTTYVSIZANCNFSM4TTG7FRA> .

[roysG]

filebeat version 7.13.3 (amd64), libbeat 7.13.3 [3ddad4c built 2021-07-02 12:11:38 +0000 UTC]

[jlind23]

Backlog grooming: Closing for now.