Filebeat is based on the Logstash Forwarder source code and replaces Logstash as the method to use for tailing log files and forwarding them to Logstash. This guide contains information to help you migrate from Logstash Forwarder to Filebeat.
Filebeat introduces the following major changes:
-
The config file was restructured and converted from JSON to YAML.
-
The registry file, which stores the state of the currently read files, was changed.
-
Command line options were removed and moved to the configuration file.
-
Configuration options for outputs are now inherited from libbeat. For details, see the {libbeat}/index.html[Beats Platform Reference].
-
The Beats input plugin for Logstash is required.
Filebeat requires the Beats input plugin for Logstash. For information about getting started with this plugin, see {libbeat}/getting-started.html#logstash-setup[Setting up Logstash].
In both the 1.5.x and 2.x versions of Logstash, this plugin can be loaded in parallel with the Lumberjack plugin used by the Logstash Forwarder.
If you have a large number of servers that you want to migrate from Logstash Forwarder to Filebeat, we recommend that you keep the Lumberjack plugin and load the Beats input plugin on the same Logstash instances, but set up the Beats input plugin to use a different port. After you have migrated all the machines to Filebeat, you can remove the Lumberjack plugin.
The registry file stores the state and location information that Filebeat uses to track
where it was last reading. Under Logstash Forwarder, this file was called .logstash-fowarder. For Filebeat,
the file was renamed. The name varies depending on the package type:
-
.filebeatfor.tar.gzand.tgzarchives -
/var/lib/filebeat/registryfor DEB and RPM packages -
c:\ProgramData\filebeat\registryfor the Windows zip file
For enhancement reasons, especially for Windows, the structure of the registry file has changed. This makes migrating the file complex and leads to potential errors.
Instead of migrating the registry file, we recommend that you start Filebeat on
the same host where Logstash Forwarder is running, and send the log files to a
different index. This will start indexing from scratch. If you want to start
reading at the end of all files, you can set the tail_files option in the
Filebeat configuration file to true.
Using this approach allows you to keep the old Logstash Forwarder running and then slowly migrate over to Filebeat.
Although Filebeat is based on Logstash Forwarder, Filebeat uses YAML for its configuration
file, rather than the JSON+comments language used by Logstash Forwarder. This means that you
will need to migrate your existing configuration files to use the YAML syntax. Filebeat has a main
configuration file called filebeat.yml, but Filebeat also accepts reading
multiple configuration files from a conf.d directory and has similar restrictions to Logstash Forwarder.
If you specify additional config files, you need to place them in a directory other than the directory
where the main Filebeat config file resides. You specify the location of the config files by using the
config_dir option to configure the path to the directory. In most cases, you can do a one-to-one
conversion to create a Filebeat config file for each Logstash Forwarder config file.
Before migrating your config files, we recommend that you first read the [filebeat-configuration-details] section to understand the Filebeat options.
|
Note
|
Logstash Forwarder has the option of autocompleting environment variables in the configuration file. This option currently doesn’t exist in Filebeat. |
To migrate the files section from the Logstash Forwarder configuration, create a prospectors section in the Filebeat config file. For example, assuming that you start
with this configuration in Logstash Forwarder:
# The list of files configurations
"files": [
# An array of hashes. Each hash tells what paths to watch and
# what fields to annotate on events from those paths.
{
"paths": [
"/var/log/messages",
"/var/log/*.log"
],
# A dictionary of fields to annotate on each event.
"fields": {
"type": "syslog",
"service": "apache",
"zone": "us-east-1"
}
}, {
# A path of "-" means stdin.
"paths": [ "-" ],
"fields": { "type": "stdin" }
}, {
"paths": [
"/var/log/apache/httpd-*.log"
],
"fields": { "type": "apache" }
}
]The equivalent prospectors section would look like this:
filebeat:
# List of prospectors to fetch data.
prospectors:
# Each - is a prospector. Below are the prospector specific configurations
-
paths:
- /var/log/messages
- "/var/log/*.log"
document_type: syslog (1)
fields:
service: apache
zone: us-east-1
-
input_type: stdin (2)
document_type: stdin
-
paths:
- "/var/log/apache2/httpd-*.log"
document_type: apache-
The
document_typeoption controls the outputtypefield, which is used by the Elasticsearch output to determine the document type. -
The explicit
input_typeoption was introduced to differentiate between normal files and stdin. In the future, additional types might be supported.
As you can see, apart from the new document_type and input_type options,
which were before implicitly defined via the type custom field, the remaining
options can be migrated mechanically.
The Filebeat configuration gives you more control over how each prospector behaves by allowing you to configure options that were previously global in Logstash Forwarder and set them separately for each prospector. See [filebeat-configuration-details].
Like Logstash Forwarder, Filebeat can communicate directly with Logstash.
Filebeat can also insert log entries directly
into Elasticsearch. This results in an output section that is a bit more complex, as
you can see in the following example. You’ll find, however, that you can easily
translate the Logstash part of the configuration from the equivalent Logstash Forwarder
configuration.
The following snippet shows the network section of the Logstash Forwarder configuration:
# The network section covers network configuration :)
"network": {
# A list of downstream servers listening for our messages.
# logstash-forwarder will pick one at random and only switch if
# the selected one appears to be dead or unresponsive
"servers": [ "localhost:5043" ],
# The path to your client ssl certificate (optional)
"ssl certificate": "./logstash-forwarder.crt",
# The path to your client ssl key (optional)
"ssl key": "./logstash-forwarder.key",
# The path to your trusted ssl CA file. This is used
# to authenticate your downstream server.
"ssl ca": "./logstash-forwarder.crt",
# Network timeout in seconds. This is most important for
# logstash-forwarder determining whether to stop waiting for an
# acknowledgement from the downstream server. If an timeout is reached,
# logstash-forwarder will assume the connection or server is bad and
# will connect to a server chosen at random from the servers list.
"timeout": 15
}The equivalent in Filebeat would look like this:
output:
logstash:
# The Logstash hosts. (1)
hosts:
- localhost:5043
# Network timeout in seconds.
timeout: 15
tls: (2)
# List of root certificates for HTTPS server verifications
certificate_authorities:
- ./logstash-forwarder.crt
# Certificate for TLS client authentication
certificate: ./logstash-forwarder.crt
# Client Certificate Key
certificate_key: ./logstash-forwarder.key-
When multiple hosts are defined, the default behavior in Filebeat is to pick a random host for new connections, similar to the Logstash Forwarder behavior. Filebeat can optionally do load balancing. For more details, see the [loadbalance] configuration option.
-
Note that if the
tlssection is missing, then TLS is disabled. TLS is automatically enabled when you add thetlssection. For more information about specific configuration options, see [configuration-output-tls].
With the refactoring of the configuration file, the following options were removed or renamed:
| Config Option | Action |
|---|---|
|
|
|
|
|
Both options were removed and replaced by logging options in libbeat. |
For more information about these options, see [filebeat-configuration-details].
Let’s see a simple, but complete example of a Logstash Forwarder configuration and its equivalent for Filebeat.
Logstash Forwarder configuration:
{
"files": [
{
"paths": [
"/var/log/*.log"
],
"fields": {
"type": "syslog",
"service": "test01"
}
}
],
"network": {
"servers": [ "localhost:5043" ],
}
}Filebeat configuration:
filebeat:
prospectors:
-
paths:
- "/var/log/*.log"
document_type: syslog
fields:
service: test01
output:
elasticsearch:
hosts: ["http://localhost:5043"]Most command line options available in Logstash Forwarder have been removed and
migrated to config file options. The only mandatory command line option for
running Filebeat is -c followed by the path to the config file. If you used command line
options with Logstash Forwarder, make sure that you add your options to the
configuration file. For naming changes, see Renamed Options.
Filebeat does provide command line options that are common to all Beats. For more details about these options, see [filebeat-command-line].
The following command line options have been renamed and moved to the config file. Also see Changed Configuration File Options for a list of configuration file options that were completely removed or replaced by options specified in libbeat.
| Command Line Option | Config File Option | Description |
|---|---|---|
|
|
The config option was split into two parts. You use the The |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
In the default configuration, Filebeat structures its output documents a little differently from the Logstash Forwarder. This section discusses the differences and the options you have in case you want compatibility with the Logstash Forwarder.
The custom fields (added from the configuration file) are set as top-level
fields in Logstash Forwarder but are grouped together under a fields
dictionary in Filebeat. If you need the old behavior during the migration phase,
you can use the [fields-under-root] configuration option:
filebeat:
prospectors:
-
paths:
- "/var/log/*.log"
document_type: syslog
fields:
service: test01
fields_under_root: trueWhile the Logstash Forwarder sends the hostname of the server it’s running on in
the host field, Filebeat uses the beat.hostname field for the same purpose.
Because host is commonly used in the Logstash plugin ecosystem, the Beats
input plugin automatically copies beat.hostname into host.
The file field was renamed to source. If you rely on this field being
named file, you can rename it by using the mutate filter in Logstash. For
example:
filter {
mutate {
rename => {
"source" => "file"
}
}
}The following list of implementation changes should not affect your experience migrating from Logstash Forwarder, but you should be aware of the changes. Please post GitHub issues if you notice any regressions from Logstash Forwarder.
The packaging process for Filebeat uses the Beats infrastructure, so some aspects of packaging, such as the init scripts, are different from Logstash Forwarder. Please post GitHub issues if you hit any issues with the new packages.
One notable change is the name of the registry file. The name varies depending on the package type:
-
.filebeatfor.tar.gzand.tgzarchives -
/usr/lib/filebeat/registryfor DEB and RPM packages -
c:\ProgramData\filebeat\registryfor the Windows zip file
Behind the scenes, Filebeat uses a sightly improved protocol for communicating with Logstash.
If you follow the section on migrating the configuration, you will have TLS
enabled. However, you must be aware that if the tls section is missing from the
configuration file, Filebeat uses an unencrypted connection to talk to Logstash.