Meta ticket: Filebeat modules

[tsg]

This is the meta ticket for the Filebeat modules implementation.

TODOs and progress:

• Prototype Filebeat modules implementation #3158 Add a sample module (NGINX)
• Prototype Filebeat modules implementation #3158 Prototype module loading
• Add support for multiple paths per fileset #3195 Add support for multiple paths on the same OS in the Nginx module
• Filebeat mysql module #3171 Add sample module for Mysql
• Filebeat syslog module #3191 Add sample module for Syslog
• Filebeat module system/integration tests #3214 Add system tests for the modules
• Moved Kibana dashboards in the module folders #3221 Move Kibana dashboards at the module level
• Generator for a Filebeat module/fileset #3248 Add a module generator
• Filebeat: Apache2 module #3256 Apache2 module
• Filebeat modules: first phase of the Go implementation #3333 Phase 1 of the go implementation: create prospector from the modules
• Load Filebeat modules pipelines on -setup #3394 Load the pipeline automatically
• Per prospector configurable pipeline #3433 Make the pipeline configurable from the prospector config
• Set the pipeline ID in the prospector configuration #3472 Pass the pipeline ID from modules automatically
• Deploy Filebeat modules in the packages #3436 Include the module files in the filebeat packages
• Make -setup load the Beat dashboards #3506 Replace the import_dashboard program with -setup
• Use the Beat version in the Ingest Node pipeline #3516 Add the Beat version in the pipeline ID
• Check if the pipeline exists before loading #3522 Only insert the pipeline if it's not already loaded
• Update Windows paths for chocolatey installed packages #3524 Test the windows versions of the Filebeat modules
• Add fileset.name and fileset.module fields #3540 Replace fields.source_type with fileset.module and fileset.name
• Added Filebeat Module overview and tutorial #3592 Docs: overview & tutorial
• Per module docs in Filebeat #3598 Docs: create & gather per module docs and configuration samples
• Filebeat modules dev guide #3616 Docs: Add a guide for creating modules
• Docs: Add Logstash equivalent configurations. Tracked in: Logstash docs and configs for Filebeat modules logstash#6542

Overview

Filebeat modules are "packages" of the required configurations and logic to ship and analyze log files from common services. A typical module (e.g nginx) is composed of several filesets, one for each type of logs (e.g. access and error for nginx). A typical fileset contains the following:

• filebeat prospector configuration
• Elasticsearch Ingest Node pipeline definition
• Fields definitions and docs
• Kibana dashboards (at the module level)
• Test log files
• A manifest.yml file with overwritable variables and logic to select the right files

At the moment, Filebeat modules are strictly configuration files and templates, no actual Go code. This makes it easy to create new module. Eventually we might have some of the modules include Go code/plugins for more specific needs.

Filebeat has code to evaluate the variables from the manifest.yml file, interpret the templates, load the Ingest Node pipeline into Elasticsearch, and the Kibana dashboards into Kibana.

User interaction / tutorial

The easiest way to ship & parse the Nginx logs would be to start Filebeat like this:

filebeat -e -modules=nginx -setup

(The -e is for sending the output to stdout)

The -setup flag instructs Filebeat to load the Kibana dashboards on startup. After that is done and you can see data in Kibana, you can restart Filebeat without the -setup flag:

filebeat -e -modules=nginx

You can also start multiple modules at once:

filebeat -e -modules=nginx,mysql,system

If you prefer the configuration file, you can add the following to it, which is the equivalent of the above:

modules:
- name: nginx
- name: mysql
- name: syslog

Then start Filebeat simply with filebeat -e.

Variable overrides

Each fileset has a set of "variables" defined in the manifest.yml file, which allow a first level of configuring the module. For example, most modules allow setting custom paths where to find the log files. For example, to adjust where the access log are you can type:

filebeat -e -modules nginx -M "nginx.access.var.paths=[/opt/apache2/logs/access.log*]"

Or via the configuration file:

modules:
- name: nginx
  access:
    var.paths = ["/opt/apache2/logs/access.log*"]

Advanced settings

Behind the scenes, each module starts a Filebeat prospector. For advanced users, it's possible to add or overwrite any of the prospector settings. For example, enabling close_eof can be done like this:

modules:
- name: nginx
  access:
    prospector.close_eof: true

Or like this:

filebeat -e -modules=nginx -M "nginx.access.prospector.close_eof=true"

From the CLI, it's possible to change variables or settings for multiple modules/fileset at once. For example, the following works and will enable close_eof for all the filesets in the nginx module:

filebeat -e -modules=nginx -M "nginx.*.prospector.close_eof=true"

The following also works and will enable close_eof for all prospectors created by modules:

filebeat -e -modules=nginx,mysql -M "*.*.prospector.close_eof=true"

[mjpowersjr]

Nice work on parsing the Mysql slow query log! I thought I would share a alternative approach to ingesting the slow query log in case it's helpful - https://gist.github.com/mjpowersjr/740a9583e9ec8b49e0a3

[tsg]

@mjpowersjr thanks for the hint, we could add the CSV format as an option to the module. That's actually a good example for a feature I was planning.

[blacktop]

Are you going to expose this like community beats?

It would be really cool to have a cookiecutter to create filebeat modules that could be installed like elasticsearch plugins?

[tsg]

@blacktop we do plan to support modules that live outside the beats repository. Not sure if it's going to be in the first version, though.

[tsg]

First phase, which we'll include as beta in 5.3, should be pretty much done.