Showing with 456 additions and 78 deletions.
  1. +17 −4 CHANGELOG.md
  2. +1 −0 Gemfile
  3. +55 −21 README.md
  4. +6 −4 manifests/config.pp
  5. +50 −12 manifests/init.pp
  6. +1 −0 manifests/install/windows.pp
  7. +2 −2 manifests/params.pp
  8. +73 −0 manifests/processor.pp
  9. +3 −0 manifests/repo.pp
  10. +1 −1 manifests/service.pp
  11. +1 −5 metadata.json
  12. +145 −0 spec/classes/filebeat_spec.rb
  13. +73 −0 spec/defines/processor_spec.rb
  14. +28 −29 templates/filebeat5.yml.erb
21 changes: 17 additions & 4 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,16 +4,29 @@ Changelog
## Unreleased
[Full Changelog](https://github.com/pcfens/puppet-filebeat/compare/v0.10.4...HEAD)

## [v0.11.0](https://github.com/pcfens/puppet-filebeat/tree/v0.11.0)
[Full Changelog](https://github.com/pcfens/puppet-filebeat/compare/v0.10.4...v0.11.0)

**Breaking Changes**
- Processors are managed by their own defined resource (slight syntax change) [\#85](https://github.com/pcfens/puppet-filebeat/pull/85)
- The registry file has likely moved because of an upstream change. Moving this file
can cause problems (duplicate or missed log entries), so you may want to point it
to your existing file (check in /.filebeat on Linux systems)

**Normal Enhancements/Bugfixes**
- Support proxy for windows file downloads [\#90](https://github.com/pcfens/puppet-filebeat/pull/90)
- Setting `package_ensure` to absent removes puppet managed files and removes the package
- Add support for index conditional output to elasticsearch [\#97](https://github.com/pcfens/puppet-filebeat/pull/97)
- Add support for a conditional pipeline for elasticsearch [\#98](https://github.com/pcfens/puppet-filebeat/pull/98)
- Template should check for nil instead of undef [\#63](https://github.com/pcfens/puppet-filebeat/issues/63)
- Support for the round_robin and group_events parameters in kafka outputs [\#100](https://github.com/pcfens/puppet-filebeat/pull/100)

## [v0.10.4](https://github.com/pcfens/puppet-filebeat/tree/v0.10.4)
[Full Changelog](https://github.com/pcfens/puppet-filebeat/compare/v0.10.3...v0.10.4)

- Add output.console support to the config template [\#91](https://github.com/pcfens/puppet-filebeat/issues/91)
- Support puppet with strict variables enabled [\#92](https://github.com/pcfens/puppet-filebeat/issues/92)

Processor support is going to change in the next feature release (v0.11.x) in a way that is not
backwards compatible with the current method. The upcoming change is documented in
[\#85](https://github.com/pcfens/puppet-filebeat/pull/85).

## [v0.10.3](https://github.com/pcfens/puppet-filebeat/tree/v0.10.3)
[Full Changelog](https://github.com/pcfens/puppet-filebeat/compare/v0.10.2...v0.10.3)

Expand Down
1 change: 1 addition & 0 deletions Gemfile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ group :test do
gem 'puppet-lint-variable_contains_upcase', :require => false
gem 'metadata-json-lint', :require => false
gem 'puppet-strings', '~> 0.99.0', :require => false
gem 'rubocop', '~> 0.47.1', :require => false
gem 'rubocop-rspec', '~> 1.6', :require => false if RUBY_VERSION >= '2.3.0'
gem 'json_pure', '<= 2.0.1', :require => false if RUBY_VERSION < '2.0.0'
gem 'mocha', '>= 1.2.1', :require => false
Expand Down
76 changes: 55 additions & 21 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,10 @@

[![Build Status](https://travis-ci.org/pcfens/puppet-filebeat.svg?branch=master)](https://travis-ci.org/pcfens/puppet-filebeat)

## Upgrading from <= v 0.10.4
Be sure to read the changelog as there are up to 2 breaking changes introduced in v0.11.0 related
to processors and the default registry path

#### Table of Contents

1. [Description](#description)
Expand All @@ -16,6 +20,7 @@
- [JSON logs](#json-logs)
- [Prospectors in hiera](#prospectors-in-hiera)
- [Usage on Windows](#usage-on-windows)
- [Processors](#processors)
4. [Reference](#reference)
- [Public Classes](#public-classes)
- [Private Classes](#private-classes)
Expand Down Expand Up @@ -154,55 +159,64 @@ but is expected to exist as a directory that puppet can write to.
### Processors

Filebeat 5.0 and greater includes a new libbeat feature for filtering and/or enhancing all
exported data through processors before geing sent to the configured output(s). By populating
the `processors` parameter any number of processors may be configured to work on all events
or only those that match certain conditions.
exported data through processors before geing sent to the configured output(s). They can be
defined as a hash added to the class declaration (also used for automatically creating
processors using hiera), or as their own defined resources.

To drop the offset and input_type fields from all events:

```puppet
class{"filebeat":
processors => [
{
"name" => "drop_fields",
processors => {
"drop_fields" => {
"params" => {"fields" => ["input_type", "offset"]}
},
],
},
}
```

To drop all events that have the http response code equal to 200:

```puppet
class{"filebeat":
processors => [
{
"name" => "drop_event",
processors => {
"drop_event" => {
"when" => {"equals" => {"http.code" => 200}}
},
],
},
}
```

Now to combine these examples into a single definition:

```puppet
class{"filebeat":
processors => [
{
"name" => "drop_fields",
"params" => {"fields" => ["input_type", "offset"]}
processors => {
"drop_fields" => {
"params" => {"fields" => ["input_type", "offset"]},
"priority" => 1,
},
{
"name" => "drop_event",
"when" => {"equals" => {"http.code" => 200}}
"drop_event" => {
"when" => {"equals" => {"http.code" => 200}},
"priority: => 2,
},
],
},
}
```

For more information please review the documentation [here](https://www.elastic.co/guide/en/beats/filebeat/5.1/configuration-processors.html).

#### Processors in Hiera

Processors can be declared in hiera using the `processors` parameter. By default, hiera will not merge
processor declarations down the hiera hierarchy. To change the behavior in puppet 3 use the `processors_merge`
parameter. In puppet 4, you can use `processors_merge`, but can also use the
[lookup_options](https://docs.puppet.com/puppet/latest/reference/lookup_quick.html#setting-lookupoptions-in-data)
flag.

When `processors_merge` is set to true, `processors` will be replaced by the output of
`hiera_hash('filebeat::processors')`.

## Reference
- [**Public Classes**](#public-classes)
- [Class: filebeat](#class-filebeat)
Expand All @@ -216,6 +230,7 @@ For more information please review the documentation [here](https://www.elastic.
- [Class: filebeat::install::windows](#class-filebeatinstallwindows)
- [**Public Defines**](#public-defines)
- [Define: filebeat::prospector](#define-filebeatprospector)
- [Define: filebeat::processors](#define-filebeatprocessor)

### Public Classes

Expand All @@ -225,15 +240,17 @@ Installs and configures filebeat.

**Parameters within `filebeat`**
- `major_version`: [String] The major version of filebeat to install. Should be either undef, 1, or 5. (default 5 if 1 not already installed)
- `package_ensure`: [String] The ensure parameter for the filebeat package (default: present)
- `package_ensure`: [String] The ensure parameter for the filebeat package If set to absent,
prospectors and processors passed as parameters are ignored and everything managed by
puppet will be removed. (default: present)
- `manage_repo`: [Boolean] Whether or not the upstream (elastic) repo should be configured or not (default: true)
- `service_ensure`: [String] The ensure parameter on the filebeat service (default: running)
- `service_enable`: [String] The enable parameter on the filebeat service (default: true)
- `service_provider`: [String] The provider parameter on the filebeat service (default: on RedHat based systems use redhat, otherwise undefined)
- `spool_size`: [Integer] How large the spool should grow before being flushed to the network (default: 2048)
- `idle_timeout`: [String] How often the spooler should be flushed even if spool size isn't reached (default: 5s)
- `publish_async`: [Boolean] If set to true filebeat will publish while preparing the next batch of lines to transmit (defualt: false)
- `registry_file`: [String] The registry file used to store positions, absolute or relative to working directory (default .filebeat)
- `registry_file`: [String] The registry file used to store positions, must be an absolute path (default is OS dependent - see params.pp)
- `config_file`: [String] Where the configuration file managed by this module should be placed. If you think
you might want to use this, read the [limitations](#using-config_file) first. Defaults to the location
that filebeat expects for your operating system.
Expand Down Expand Up @@ -332,6 +349,23 @@ to fully understand what these parameters do.
- `multiline`: [Hash] Options that control how Filebeat handles log messages that span multiple lines.
[See above](#multiline-logs). (default: {})

#### Define: `filebeat::processor`

Installs a configuration file for a processor.

Be sure to read the [processor configuration details](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-processors.html)
to fully understand what these parameters do.

**Parameters for `filebeat::processor`**
- `ensure`: The ensure parameter on the prospector configuration file. (default: present)
- `priority`: [Integer] Used to help alpha-numerically sort the processor files in the config dir
location. (default: 10)
- `processor_name`: [String] The name of the processor. (default: $name)
- `params`: [Hash] The key-value pairs to pass to the processor as parameters. Not required on all
processors, review documentation for more details. (default: undef)
- `when`: Optional[Hash] Run this processor on any event that matches these conditions. Populates
the `when` option. (default: undef)


## Limitations
This module doesn't load the [elasticsearch index template](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-getting-started.html#filebeat-template) into elasticsearch (required when shipping
Expand Down
10 changes: 6 additions & 4 deletions manifests/config.pp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@
}

file {'filebeat.yml':
ensure => file,
ensure => $filebeat::file_ensure,
path => $filebeat::config_file,
content => template($filebeat::real_conf_template),
owner => 'root',
Expand All @@ -45,21 +45,22 @@
}

file {'filebeat-config-dir':
ensure => directory,
ensure => $filebeat::directory_ensure,
path => $filebeat::config_dir,
owner => 'root',
group => 'root',
mode => $filebeat::config_dir_mode,
recurse => $filebeat::purge_conf_dir,
purge => $filebeat::purge_conf_dir,
force => true,
}
} # end Linux

'Windows' : {
$filebeat_path = 'c:\Program Files\Filebeat\filebeat.exe'

file {'filebeat.yml':
ensure => file,
ensure => $filebeat::file_ensure,
path => $filebeat::config_file,
content => template($filebeat::real_conf_template),
validate_cmd => "\"${filebeat_path}\" -N -configtest -c \"%\"",
Expand All @@ -68,10 +69,11 @@
}

file {'filebeat-config-dir':
ensure => directory,
ensure => $filebeat::directory_ensure,
path => $filebeat::config_dir,
recurse => $filebeat::purge_conf_dir,
purge => $filebeat::purge_conf_dir,
force => true,
}
} # end Windows

Expand Down
62 changes: 50 additions & 12 deletions manifests/init.pp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -42,9 +42,10 @@
# @param max_procs [Number] The maximum number of CPUs that can be simultaneously used
# @param fields [Hash] Optional fields that should be added to each event output
# @param fields_under_root [Boolean] If set to true, custom fields are stored in the top level instead of under fields
# @param processors [Array] An optional list of hashes used to configure filebeat processors
# @param processors [Hash] Processors that will be added. Commonly used to create processors using hiera.
# @param prospectors [Hash] Prospectors that will be created. Commonly used to create prospectors using hiera
# @param prospectors_merge [Boolean] Whether $prospectors should merge all hiera sources, or use simple automatic parameter lookup
# proxy_address [String] Proxy server to use for downloading files
class filebeat (
$major_version = undef,
$package_ensure = $filebeat::params::package_ensure,
Expand Down Expand Up @@ -78,17 +79,19 @@
$max_procs = $filebeat::params::max_procs,
$fields = $filebeat::params::fields,
$fields_under_root = $filebeat::params::fields_under_root,
$processors = $filebeat::params::processors,
$processors = {},
$processors_merge = false,
#### End v5 onlly ####
$prospectors = {},
$prospectors_merge = false,
$proxy_address = undef
) inherits filebeat::params {

include ::stdlib

$kernel_fail_message = "${::kernel} is not supported by filebeat."

validate_bool($manage_repo, $prospectors_merge)
validate_bool($manage_repo, $processors_merge, $prospectors_merge)

if $major_version == undef and getvar('::filebeat_version') == undef {
$real_version = '5'
Expand Down Expand Up @@ -116,32 +119,67 @@
}
}


if $prospectors_merge {
$prospectors_final = hiera_hash('filebeat::prospectors', $prospectors)
} else {
$prospectors_final = $prospectors
}

if $processors_merge {
$processors_final = hiera_hash('filebeat::processors', $processors)
} else {
$processors_final = $processors
}

if $config_file != $filebeat::params::config_file {
warning('You\'ve specified a non-standard config_file location - filebeat may fail to start unless you\'re doing something to fix this')
}

validate_array($processors)
validate_hash($outputs, $logging, $prospectors_final)
validate_string($idle_timeout, $registry_file, $config_dir, $package_ensure)

if $package_ensure == '1.0.0-beta4' or $package_ensure == '1.0.0-rc1' {
fail('Filebeat versions 1.0.0-rc1 and before are unsupported because they don\'t parse normal YAML headers')
}

anchor { 'filebeat::begin': } ->
class { '::filebeat::install': } ->
class { '::filebeat::config': } ->
class { '::filebeat::service': } ->
anchor { 'filebeat::end': }
if(!empty($proxy_address)){
validate_re($proxy_address, ['^(http(?:s)?\:\/\/[a-zA-Z0-9]+(?:(?:\.|\-)[a-zA-Z0-9]+)+(?:\:\d+)?(?:\/[\w\-]+)*(?:\/?|\/\w+\.[a-zA-Z]{2,4}(?:\?[\w]+\=[\w\-]+)?)?(?:\&[\w]+\=[\w\-]+)*)$'], 'ERROR: You must enter a proxy url in a valid format i.e. http://proxy.net:3128')
}

if !empty($prospectors_final) {
create_resources('filebeat::prospector', $prospectors_final)
if $package_ensure == 'absent' {
$alternate_ensure = 'absent'
$real_service_ensure = 'stopped'
$file_ensure = 'absent'
$directory_ensure = 'absent'
} else {
$alternate_ensure = 'present'
$file_ensure = 'file'
$directory_ensure = 'directory'
$real_service_ensure = $service_ensure
}

# If we're removing filebeat, do things in a different order to make sure
# we remove as much as possible
if $package_ensure == 'absent' {
anchor { 'filebeat::begin': }
-> class { '::filebeat::config': }
-> class { '::filebeat::install': }
-> class { '::filebeat::service': }
-> anchor { 'filebeat::end': }
} else {
anchor { 'filebeat::begin': }
-> class { '::filebeat::install': }
-> class { '::filebeat::config': }
-> class { '::filebeat::service': }
-> anchor { 'filebeat::end': }
}

if $package_ensure != 'absent' {
if !empty($prospectors_final) {
create_resources('filebeat::prospector', $prospectors_final)
}
if !empty($processors_final) {
create_resources('filebeat::processor', $processors_final)
}
}
}
1 change: 1 addition & 0 deletions manifests/install/windows.pp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
ensure => present,
source => $filebeat::download_url,
verify_peer => false,
proxy => $filebeat::proxy_address,
}

exec { "unzip ${filename}":
Expand Down
4 changes: 2 additions & 2 deletions manifests/params.pp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,6 @@
$tags = []
$queue_size = 1000
$max_procs = undef
$registry_file = '.filebeat'
$config_dir_mode = '0755'
$config_file_mode = '0644'
$purge_conf_dir = true
Expand All @@ -22,7 +21,6 @@
$logging = {}
$run_options = {}
$use_generic_template = false
$processors = []

# These are irrelevant as long as the template is set based on the major_version parameter
# if versioncmp('1.9.1', $::rubyversion) > 0 {
Expand All @@ -35,6 +33,7 @@
'Linux' : {
$config_file = '/etc/filebeat/filebeat.yml'
$config_dir = '/etc/filebeat/conf.d'
$registry_file = '/var/lib/filebeat/registry'

# These parameters are ignored if/until tarball installs are supported in Linux
$tmp_dir = '/tmp'
Expand All @@ -53,6 +52,7 @@
'Windows' : {
$config_file = 'C:/Program Files/Filebeat/filebeat.yml'
$config_dir = 'C:/Program Files/Filebeat/conf.d'
$registry_file = 'C:/ProgramData/filebeat/registry'
$download_url = 'https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.1.1-windows-x86_64.zip'
$install_dir = 'C:/Program Files'
$tmp_dir = 'C:/Windows/Temp'
Expand Down
Loading