Showing with 193 additions and 204 deletions.
  1. +0 −3 .travis.yml
  2. +10 −1 CHANGELOG.md
  3. +32 −40 README.md
  4. +2 −4 manifests/config.pp
  5. +10 −12 manifests/init.pp
  6. +132 −0 manifests/input.pp
  7. +4 −8 manifests/params.pp
  8. +0 −132 manifests/prospector.pp
  9. +1 −1 metadata.json
  10. +1 −1 spec/defines/{prospector_spec.rb → input_spec.rb}
  11. +0 −1 templates/filebeat.yml.erb
  12. +1 −1 templates/prospector.yml.erb
3 changes: 0 additions & 3 deletions .travis.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,6 @@ matrix:
-
env: PUPPET_GEM_VERSION="~> 5.0" CHECK=parallel_spec
rvm: 2.4.4
-
env: PUPPET_GEM_VERSION="~> 4.0" CHECK=parallel_spec
rvm: 2.1.9
notifications:
email: false
deploy:
Expand Down
11 changes: 10 additions & 1 deletion CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,16 @@ Changelog
=========

## Unreleased
[Full Changelog](https://github.com/pcfens/puppet-filebeat/compare/v3.4.0...HEAD)
[Full Changelog](https://github.com/pcfens/puppet-filebeat/compare/v4.0.0...HEAD)

## [v4.0.0](https://github.com/pcfens/puppet-filebeat/tree/v4.0.0)
[Full Changelog](https://github.com/pcfens/puppet-filebeat/compare/v3.4.0...v4.0.0)

- Switch from `filebeat::prospector` to `filebeat::input` to reflect the changes
in the upstream filebeat configuration.
- Add support for Filebeat 7
- Remove support for `registry_file` and `registry_flush` settings (removed in 7.x)
- Remove `queue_size` parameter

## [v3.4.0](https://github.com/pcfens/puppet-filebeat/tree/v3.4.0)
[Full Changelog](https://github.com/pcfens/puppet-filebeat/compare/v3.3.3...v3.4.0)
Expand Down
72 changes: 32 additions & 40 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,10 @@
- [Setup requirements](#setup-requirements)
- [Beginning with filebeat](#beginning-with-filebeat)
3. [Usage - Configuration options and additional functionality](#usage)
- [Adding a prospector](#adding-a-prospector)
- [Adding an Input](#adding-an-input)
- [Multiline Logs](#multiline-logs)
- [JSON logs](#json-logs)
- [Prospectors in hiera](#prospectors-in-hiera)
- [Inputs in hiera](#inputs-in-hiera)
- [Usage on Windows](#usage-on-windows)
- [Processors](#processors)
4. [Reference](#reference)
Expand All @@ -36,9 +36,12 @@ The `filebeat` module installs and configures the [filebeat log shipper](https:/
By default `filebeat` adds a software repository to your system, and installs filebeat along
with required configurations.

### Upgrading to Filebeat 6.x
### Upgrading to Filebeat 7.x

To upgrade to Filebeat 6.x, simply set `$filebeat::major_version` to `6` and `$filebeat::package_ensure` to `latest` (or whichever version of 6.x you want, just not present).
To upgrade to Filebeat 7.x, simply set `$filebeat::major_version` to `6` and `$filebeat::package_ensure` to `latest` (or whichever version of 7.x you want, just not present).

You'll also need to change instances of `filebeat::prospector` to `filebeat::input` when upgrading to version 4.x of
this module.


### Setup Requirements
Expand Down Expand Up @@ -95,18 +98,18 @@ class { 'filebeat':
[logging](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-configuration-details.html#configuration-logging) options
can be configured the same way, and are documented on the [elastic website](https://www.elastic.co/guide/en/beats/filebeat/current/index.html).

### Adding a prospector
### Adding an Input

Prospectors are processes that ship log files to elasticsearch or logstash. They can
Inputs are processes that ship log files to elasticsearch or logstash. They can
be defined as a hash added to the class declaration (also used for automatically creating
prospectors using hiera), or as their own defined resources.
input using hiera), or as their own defined resources.

At a minimum, the `paths` parameter must be set to an array of files or blobs that should
be shipped. `doc_type` is what logstash views as the type parameter if you'd like to
apply conditional filters.

```puppet
filebeat::prospector { 'syslogs':
filebeat::input { 'syslogs':
paths => [
'/var/log/auth.log',
'/var/log/syslog',
Expand All @@ -117,20 +120,20 @@ filebeat::prospector { 'syslogs':

#### Multiline Logs

Filebeat prospectors can handle multiline log entries. The `multiline`
Filebeat inputs can handle multiline log entries. The `multiline`
parameter accepts a hash containing `pattern`, `negate`, `match`, `max_lines`, and `timeout`
as documented in the filebeat [configuration documentation](https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html).

#### JSON Logs

Filebeat prospectors (versions >= 5.0) can natively decode JSON objects if they are stored one per line. The `json`
Filebeat inputs (versions >= 5.0) can natively decode JSON objects if they are stored one per line. The `json`
parameter accepts a hash containing `message_key`, `keys_under_root`, `overwrite_keys`, and `add_error_key`
as documented in the filebeat [configuration documentation](https://www.elastic.co/guide/en/beats/filebeat/5.5/configuration-filebeat-options.html#config-json).

### Prospectors in Hiera
### Inputs in Hiera

Prospectors can be defined in hiera using the `prospectors` parameter. By default, hiera will not merge
prospector declarations down the hiera hierarchy. That behavior can be changed by configuring the
Inputs can be defined in hiera using the `inputs` parameter. By default, hiera will not merge
input declarations down the hiera hierarchy. That behavior can be changed by configuring the
[lookup_options](https://docs.puppet.com/puppet/latest/reference/lookup_quick.html#setting-lookupoptions-in-data)
flag.

Expand Down Expand Up @@ -161,7 +164,7 @@ class{"filebeat":
```

To drop all events that have the http response code equal to 200:

input
```puppet
class{"filebeat":
processors => {
Expand Down Expand Up @@ -210,7 +213,7 @@ flag.
- [Class: filebeat::install::linux](#class-filebeatinstalllinux)
- [Class: filebeat::install::windows](#class-filebeatinstallwindows)
- [**Public Defines**](#public-defines)
- [Define: filebeat::prospector](#define-filebeatprospector)
- [Define: filebeat::input](#define-filebeatinput)
- [Define: filebeat::processors](#define-filebeatprocessor)

### Public Classes
Expand All @@ -221,7 +224,7 @@ Installs and configures filebeat.

**Parameters within `filebeat`**
- `package_ensure`: [String] The ensure parameter for the filebeat package If set to absent,
prospectors and processors passed as parameters are ignored and everything managed by
inputs and processors passed as parameters are ignored and everything managed by
puppet will be removed. (default: present)
- `manage_repo`: [Boolean] Whether or not the upstream (elastic) repo should be configured or not (default: true)
- `major_version`: [Enum] The major version of Filebeat to install. Should be either `'5'` or `'6'`. The default value is `'6'`, except
Expand All @@ -233,18 +236,17 @@ Installs and configures filebeat.
- `spool_size`: [Integer] How large the spool should grow before being flushed to the network (default: 2048)
- `idle_timeout`: [String] How often the spooler should be flushed even if spool size isn't reached (default: 5s)
- `publish_async`: [Boolean] If set to true filebeat will publish while preparing the next batch of lines to transmit (default: false)
- `registry_file`: [String] The registry file used to store positions, must be an absolute path (default is OS dependent - see params.pp)
- `config_file`: [String] Where the configuration file managed by this module should be placed. If you think
you might want to use this, read the [limitations](#using-config_file) first. Defaults to the location
that filebeat expects for your operating system.
- `config_dir`: [String] The directory where prospectors should be defined (default: /etc/filebeat/conf.d)
- `config_dir`: [String] The directory where inputs should be defined (default: /etc/filebeat/conf.d)
- `config_dir_mode`: [String] The permissions mode set on the configuration directory (default: 0755)
- `config_dir_owner`: [String] The owner of the configuration directory (default: root). Linux only.
- `config_dir_group`: [String] The group of the configuration directory (default: root). Linux only.
- `config_file_mode`: [String] The permissions mode set on configuration files (default: 0644)
- `config_file_owner`: [String] The owner of the configuration files, including prospectors (default: root). Linux only.
- `config_file_group`: [String] The group of the configuration files, including prospectors (default: root). Linux only.
- `purge_conf_dir`: [Boolean] Should files in the prospector configuration directory not managed by puppet be automatically purged
- `config_file_owner`: [String] The owner of the configuration files, including inputs (default: root). Linux only.
- `config_file_group`: [String] The group of the configuration files, including inputs (default: root). Linux only.
- `purge_conf_dir`: [Boolean] Should files in the input configuration directory not managed by puppet be automatically purged
- `enable_conf_modules`: [Boolean] Should filebeat.config.modules be enabled
- `modules_dir`: [String] The directory where module configurations should be defined (default: /etc/filebeat/modules.d)
- `outputs`: [Hash] Will be converted to YAML for the required outputs section of the configuration (see documentation, and above)
Expand All @@ -258,21 +260,20 @@ Installs and configures filebeat.
- `shutdown_timeout`: [String] How long filebeat waits on shutdown for the publisher to finish sending events
- `beat_name`: [String] The name of the beat shipper (default: hostname)
- `tags`: [Array] A list of tags that will be included with each published transaction
- `queue_size`: [String] The internal queue size for events in the pipeline
- `max_procs`: [Number] The maximum number of CPUs that can be simultaneously used
- `fields`: [Hash] Optional fields that should be added to each event output
- `fields_under_root`: [Boolean] If set to true, custom fields are stored in the top level instead of under fields
- `disable_config_test`: [Boolean] If set to true, configuration tests won't be run on config files before writing them.
- `processors`: [Hash] Processors that should be configured.
- `prospectors`: [Hash] Prospectors that will be created. Commonly used to create prospectors using hiera
- `inputs`: [Hash] Inputs that will be created. Commonly used to create inputs using hiera
- `setup`: [Hash] Setup that will be created. Commonly used to create setup using hiera
- `xpack`: [Hash] XPack configuration to pass to filebeat

### Private Classes

#### Class: `filebeat::config`

Creates the configuration files required for filebeat (but not the prospectors)
Creates the configuration files required for filebeat (but not the inputs)

#### Class: `filebeat::install`

Expand Down Expand Up @@ -300,16 +301,16 @@ Downloads, extracts, and installs the filebeat zip file in Windows.

### Public Defines

#### Define: `filebeat::prospector`
#### Define: `filebeat::input`

Installs a configuration file for a prospector.
Installs a configuration file for a input.

Be sure to read the [filebeat configuration details](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-configuration-details.html)
to fully understand what these parameters do.

**Parameters for `filebeat::prospector`**
- `ensure`: The ensure parameter on the prospector configuration file. (default: present)
- `paths`: [Array] The paths, or blobs that should be handled by the prospector. (required only if input_type is not _docker_)
**Parameters for `filebeat::input`**
- `ensure`: The ensure parameter on the input configuration file. (default: present)
- `paths`: [Array] The paths, or blobs that should be handled by the input. (required only if input_type is not _docker_)
- `containers_ids`: [Array] If input_type is _docker_, the list of Docker container ids to read the logs from. (default: '*')
- `containers_path`: [String] If input_type is _docker_, the path from where the logs should be read from. (default: /var/log/docker/containers)
- `containers_stream`: [String] If input_type is _docker_, read from the specified stream only. (default: all)
Expand All @@ -326,15 +327,15 @@ to fully understand what these parameters do.
- `log_type`: [String] \(Deprecated - use `doc_type`\) The document_type setting (optional - default: log)
- `doc_type`: [String] The event type to used for published lines, used as type field in logstash
and elasticsearch (optional - default: log)
- `scan_frequency`: [String] How often should the prospector check for new files (default: 10s)
- `scan_frequency`: [String] How often should the input check for new files (default: 10s)
- `harvester_buffer_size`: [Integer] The buffer size the harvester uses when fetching the file (default: 16384)
- `tail_files`: [Boolean] If true, filebeat starts reading new files at the end instead of the beginning (default: false)
- `backoff`: [String] How long filebeat should wait between scanning a file after reaching EOF (default: 1s)
- `max_backoff`: [String] The maximum wait time to scan a file for new lines to ship (default: 10s)
- `backoff_factor`: [Integer] `backoff` is multiplied by this parameter until `max_backoff` is reached to
determine the actual backoff (default: 2)
- `force_close_files`: [Boolean] Should filebeat forcibly close a file when renamed (default: false)
- `pipeline`: [String] Filebeat can be configured for a different ingest pipeline for each prospector (default: undef)
- `pipeline`: [String] Filebeat can be configured for a different ingest pipeline for each input (default: undef)
- `include_lines`: [Array] A list of regular expressions to match the lines that you want to include.
Ignored if empty (default: [])
- `exclude_lines`: [Array] A list of regular expressions to match the files that you want to exclude.
Expand All @@ -359,15 +360,6 @@ By default, a generic, open ended template is used that simply converts your con
a hash that is produced as YAML on the system. To use a template that is more strict, but possibly
incomplete, set `conf_template` to `filebeat/filebeat.yml.erb`.

### Registry Path

The default registry file in this module doesn't match the filebeat default, but moving the file
while the filbeat service is running can cause data duplication or data loss. If you're installing
filebeat for the first time you should consider setting `registry_file` to match the
[default](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-global-options.html#_registry_file).

Be sure to include a path or the file will be put at the root of your filesystem.

### Debian Systems

Filebeat 5.x and newer requires apt-transport-https, but this module won't install it for you.
Expand Down
6 changes: 2 additions & 4 deletions manifests/config.pp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,7 @@
'fields' => $filebeat::fields,
'fields_under_root' => $filebeat::fields_under_root,
'filebeat' => {
'registry_file' => $filebeat::registry_file,
'config.prospectors' => {
'config.inputs' => {
'enabled' => true,
'path' => "${filebeat::config_dir}/*.yml",
},
Expand All @@ -27,6 +26,7 @@
'shutdown_timeout' => $filebeat::shutdown_timeout,
'modules' => $filebeat::modules,
},
'http' => $filebeat::http,
'output' => $filebeat::outputs,
'shipper' => $filebeat::shipper,
'logging' => $filebeat::logging,
Expand Down Expand Up @@ -82,8 +82,6 @@
$skip_validation = false
}

Filebeat::Prospector <| |> -> File['filebeat.yml']

case $::kernel {
'Linux' : {
$validate_cmd = ($filebeat::disable_config_test or $skip_validation) ? {
Expand Down
Loading