10 changes: 5 additions & 5 deletions .travis.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,29 +2,29 @@
language: ruby
cache: bundler
script:
- "bundle exec rake validate lint parallel_spec SPEC_OPTS='--format documentation'"
- "bundle exec rake validate lint spec SPEC_OPTS='--format documentation'"
matrix:
fast_finish: true
include:
- rvm: 2.1.6
- rvm: 2.3.1
dist: trusty
env: PUPPET_INSTALL_TYPE=agent BEAKER_debug=true
script: bundle exec rake beaker
services: docker
sudo: required
- rvm: 2.1.6
- rvm: 2.3.1
dist: trusty
env: BEAKER_set=ubuntu-16.04 PUPPET_INSTALL_TYPE=agent BEAKER_debug=true
script: bundle exec rake beaker
services: docker
sudo: required
- rvm: 2.1.6
- rvm: 2.3.1
dist: trusty
env: BEAKER_set=centos-7 PUPPET_INSTALL_TYPE=agent BEAKER_debug=true
script: bundle exec rake beaker
services: docker
sudo: required
- rvm: 2.1.6
- rvm: 2.1.9
bundler_args: --without system_tests
env: PUPPET_GEM_VERSION="~> 4.5" STRICT_VARIABLES="yes"
- rvm: 2.1.6
Expand Down
16 changes: 16 additions & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,22 @@ Changelog
## Unreleased
[Full Changelog](https://github.com/pcfens/puppet-filebeat/compare/v0.7.4...HEAD)

## [v0.8.0](https://github.com/pcfens/puppet-filebeat/tree/v0.8.0)
[Full Changelog](https://github.com/pcfens/puppet-filebeat/compare/v0.7.4...v0.8.0)

**Enhancements**
- Add support for Filebeat v5.

If you use this module on a system with filebeat 1.x installed, and you keep your current parameters
nothing will change. Setting `major_version` to '5' will modify the configuration template and update
package repositories, but won't update the package itself. To update the package set the
`package_ensure` parameter to at least 5.0.0.

- Add a parameter `use_generic_template` that uses a more generic version of the configuration
template. The generic template is more future proof (if types are correct), but looks
very different than the example file.


## [v0.7.4](https://github.com/pcfens/puppet-filebeat/tree/v0.7.4)
[Full Changelog](https://github.com/pcfens/puppet-filebeat/compare/v0.7.2...v0.7.4)

Expand Down
1 change: 0 additions & 1 deletion Gemfile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@ gem 'json', :require => false
gem 'puppetlabs_spec_helper', :require => false
gem 'rspec-puppet', '>= 2.3.2', :require => false
gem 'simplecov', :require => false
gem 'parallel_tests', :require => false
end

group :system_tests do
Expand Down
29 changes: 27 additions & 2 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,18 @@

[![Build Status](https://travis-ci.org/pcfens/puppet-filebeat.svg?branch=master)](https://travis-ci.org/pcfens/puppet-filebeat)

## Warning

Version 0.8.0 adds support for filebeat version 5. You may notice some changes to your configuration files.

The changelog has more information.

#### Table of Contents

1. [Description](#description)
2. [Setup - The basics of getting started with filebeat](#setup)
- [What filebeat affects](#what-filebeat-affects)
- [Upgrading to Filebeat 5.x](#upgrading-to-filebeat-5x)
- [Setup requirements](#setup-requirements)
- [Beginning with filebeat](#beginning-with-filebeat)
3. [Usage - Configuration options and additional functionality](#usage)
Expand Down Expand Up @@ -34,6 +41,16 @@ The `filebeat` module installs and configures the [filebeat log shipper](https:/
By default `filebeat` adds a software repository to your system, and installs filebeat along
with required configurations.

### Upgrading to Filebeat 5.x

If you use this module on a system with filebeat 1.x installed, and you keep your current parameters
nothing will change. Setting `major_version` to '5' will modify the configuration template and update
package repositories, but won't update the package itself. To update the package set the
`package_ensure` parameter to at least 5.0.0.

Windows users should set `major_version` to 5 and update the `download_url` parameter to the correct
[download](https://www.elastic.co/downloads/beats/filebeat).

### Setup Requirements

The `filebeat` module depends on [`puppetlabs/stdlib`](https://forge.puppetlabs.com/puppetlabs/stdlib), and on
Expand Down Expand Up @@ -151,6 +168,7 @@ but is expected to exist as a directory that puppet can write to.
Installs and configures filebeat.

**Parameters within `filebeat`**
- `major_version`: [String] The major version of filebeat to install. Should be either undef, 1, or 5. (default 5 if 1 not already installed)
- `package_ensure`: [String] The ensure parameter for the filebeat package (default: present)
- `manage_repo`: [Boolean] Whether or not the upstream (elastic) repo should be configured or not (default: true)
- `service_ensure`: [String] The ensure parameter on the filebeat service (default: running)
Expand All @@ -174,6 +192,15 @@ Installs and configures filebeat.
- `download_url`: [String] The URL of the zip file that should be downloaded to install filebeat (windows only)
- `install_dir`: [String] Where filebeat should be installed (windows only)
- `tmp_dir`: [String] Where filebeat should be temporarily downloaded to so it can be installed (windows only)
- `use_generic_template`: [Boolean] Use a more generic version of the configuration template. The generic template is more
future proof (if types are correct), but looks very different than the example file (default: false)
- `shutdown_timeout`: [String] How long filebeat waits on shutdown for the publisher to finish sending events
- `beat_name`: [String] The name of the beat shipper (default: hostname)
- `tags`: [Array] A list of tags that will be included with each published transaction
- `queue_size`: [String] The internal queue size for events in the pipeline
- `max_procs`: [Number] The maximum number of CPUs that can be simultaneously used
- `fields`: [Hash] Optional fields that should be added to each event output
- `fields_under_root`: [Boolean] If set to true, custom fields are stored in the top level instead of under fields
- `prospectors`: [Hash] Prospectors that will be created. Commonly used to create prospectors using hiera
- `prospectors_merge`: [Boolean] If true, `hiera_hash()` will be used to build the the `prospectors` parameter (default: false)

Expand Down Expand Up @@ -238,8 +265,6 @@ to fully understand what these parameters do.
- `max_backoff`: [String] The maximum wait time to scan a file for new lines to ship (default: 10s)
- `backoff_factor`: [Integer] `backoff` is multiplied by this parameter until `max_backoff` is reached to
determine the actual backoff (default: 2)
- `partial_line_waiting`: [String] How long should the prospector wait before shipping a file with
a potentially incomplete last line (default: 5s)
- `force_close_files`: [Boolean] Should filebeat forcibly close a file when renamed (default: false)
- `include_lines`: [Array] A list of regular expressions to match the lines that you want to include.
Ignored if empty (default: [])
Expand Down
2 changes: 0 additions & 2 deletions Rakefile
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
require 'puppet-lint/tasks/puppet-lint'
require 'puppetlabs_spec_helper/rake_tasks'
require 'parallel_tests'
require 'parallel_tests/cli'

PuppetLint.configuration.fail_on_warnings = true
PuppetLint.configuration.send('relative')
Expand Down
14 changes: 14 additions & 0 deletions lib/facter/filebeat_version.rb
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
require 'facter'
Facter.add('filebeat_version') do
confine :kernel => %w(Linux Windows)
setcode do
if File.executable?('/usr/bin/filebeat')
filebeat_version = Facter::Util::Resolution.exec('/usr/bin/filebeat --version')
elsif File.executable?('/usr/share/filebeat/bin/filebeat')
filebeat_version = Facter::Util::Resolution.exec('/usr/share/filebeat/bin/filebeat --version')
elsif File.executable?('C:/Program Files/Filebeat/filebeat.exe --version')
filebeat_version = Facter::Util::Resolution.exec('C:/Program Files/Filebeat/filebeat.exe --version')
end
%r{^filebeat version ([^\s]+)?}.match(filebeat_version)[1] unless filebeat_version.nil?
end
end
32 changes: 20 additions & 12 deletions manifests/config.pp
View file
Original file line number Diff line number Diff line change
@@ -1,24 +1,32 @@
class filebeat::config {
$filebeat_config = {
'filebeat' => {
'spool_size' => $filebeat::spool_size,
'idle_timeout' => $filebeat::idle_timeout,
'registry_file' => $filebeat::registry_file,
'publish_async' => $filebeat::publish_async,
'config_dir' => $filebeat::config_dir,
'shutdown_timeout' => $filebeat::shutdown_timeout,
'beat_name' => $filebeat::beat_name,
'tags' => $filebeat::tags,
'queue_size' => $filebeat::queue_size,
'max_procs' => $filebeat::max_procs,
'fields' => $filebeat::fields,
'fields_under_root' => $filebeat::fields_under_root,
'filebeat' => {
'spool_size' => $filebeat::spool_size,
'idle_timeout' => $filebeat::idle_timeout,
'registry_file' => $filebeat::registry_file,
'publish_async' => $filebeat::publish_async,
'config_dir' => $filebeat::config_dir,
'shutdown_timeout' => $filebeat::shutdown_timeout,
},
'output' => $filebeat::outputs,
'shipper' => $filebeat::shipper,
'logging' => $filebeat::logging,
'runoptions' => $filebeat::run_options,
'output' => $filebeat::outputs,
'shipper' => $filebeat::shipper,
'logging' => $filebeat::logging,
'runoptions' => $filebeat::run_options,
}

case $::kernel {
'Linux' : {
file {'filebeat.yml':
ensure => file,
path => $filebeat::config_file,
content => template($filebeat::conf_template),
content => template($filebeat::real_conf_template),
owner => 'root',
group => 'root',
mode => $filebeat::config_file_mode,
Expand All @@ -40,7 +48,7 @@
file {'filebeat.yml':
ensure => file,
path => $filebeat::config_file,
content => template($filebeat::conf_template),
content => template($filebeat::real_conf_template),
notify => Service['filebeat'],
}

Expand Down
97 changes: 73 additions & 24 deletions manifests/init.pp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@
# },
# }
#
# @param major_version [String] The major version of filebeat to install. Should be either undef, 1, or
# 5. (default 5 if 1 not already installed)
# @param package_ensure [String] The ensure parameter for the filebeat package (default: present)
# @param manage_repo [Boolean] Whether or not the upstream (elastic) repo should be configured or not (default: true)
# @param service_ensure [String] The ensure parameter on the filebeat service (default: running)
Expand All @@ -31,39 +33,86 @@
# @param download_url [String] The URL of the zip file that should be downloaded to install filebeat (windows only)
# @param install_dir [String] Where filebeat should be installed (windows only)
# @param tmp_dir [String] Where filebeat should be temporarily downloaded to so it can be installed (windows only)
# @param use_generic_template [Boolean] Use a more generic version of the configuration template. The generic template is more
# future proof (if types are correct), but looks very different than the example file (default: false)
# @param shutdown_timeout [String] How long filebeat waits on shutdown for the publisher to finish sending events
# @param beat_name [String] The name of the beat shipper (default: hostname)
# @param tags [Array] A list of tags that will be included with each published transaction
# @param queue_size [String] The internal queue size for events in the pipeline
# @param max_procs [Number] The maximum number of CPUs that can be simultaneously used
# @param fields [Hash] Optional fields that should be added to each event output
# @param fields_under_root [Boolean] If set to true, custom fields are stored in the top level instead of under fields
# @param prospectors [Hash] Prospectors that will be created. Commonly used to create prospectors using hiera
# @param prospectors_merge [Boolean] Whether $prospectors should merge all hiera sources, or use simple automatic parameter lookup
class filebeat (
$package_ensure = $filebeat::params::package_ensure,
$manage_repo = $filebeat::params::manage_repo,
$service_ensure = $filebeat::params::service_ensure,
$service_enable = $filebeat::params::service_enable,
$service_provider = $filebeat::params::service_provider,
$spool_size = $filebeat::params::spool_size,
$idle_timeout = $filebeat::params::idle_timeout,
$publish_async = $filebeat::params::publish_async,
$registry_file = $filebeat::params::registry_file,
$config_file = $filebeat::params::config_file,
$config_dir = $filebeat::params::config_dir,
$config_dir_mode = $filebeat::params::config_dir_mode,
$config_file_mode = $filebeat::params::config_file_mode,
$purge_conf_dir = $filebeat::params::purge_conf_dir,
$outputs = $filebeat::params::outputs,
$shipper = $filebeat::params::shipper,
$logging = $filebeat::params::logging,
$run_options = $filebeat::params::run_options,
$conf_template = $filebeat::params::conf_template,
$download_url = $filebeat::params::download_url,
$install_dir = $filebeat::params::install_dir,
$tmp_dir = $filebeat::params::tmp_dir,
$prospectors = {},
$prospectors_merge = false,
$major_version = undef,
$package_ensure = $filebeat::params::package_ensure,
$manage_repo = $filebeat::params::manage_repo,
$service_ensure = $filebeat::params::service_ensure,
$service_enable = $filebeat::params::service_enable,
$service_provider = $filebeat::params::service_provider,
$spool_size = $filebeat::params::spool_size,
$idle_timeout = $filebeat::params::idle_timeout,
$publish_async = $filebeat::params::publish_async,
$registry_file = $filebeat::params::registry_file,
$config_file = $filebeat::params::config_file,
$config_dir_mode = $filebeat::params::config_dir_mode,
$config_dir = $filebeat::params::config_dir,
$config_file_mode = $filebeat::params::config_file_mode,
$purge_conf_dir = $filebeat::params::purge_conf_dir,
$outputs = $filebeat::params::outputs,
$shipper = $filebeat::params::shipper,
$logging = $filebeat::params::logging,
$run_options = $filebeat::params::run_options,
$conf_template = undef,
$download_url = $filebeat::params::download_url,
$install_dir = $filebeat::params::install_dir,
$tmp_dir = $filebeat::params::tmp_dir,
#### v5 only ####
$use_generic_template = $filebeat::params::use_generic_template,
$shutdown_timeout = $filebeat::params::shutdown_timeout,
$beat_name = $filebeat::params::beat_name,
$tags = $filebeat::params::tags,
$queue_size = $filebeat::params::queue_size,
$max_procs = $filebeat::params::max_procs,
$fields = $filebeat::params::fields,
$fields_under_root = $filebeat::params::fields_under_root,
#### End v5 onlly ####
$prospectors = {},
$prospectors_merge = false,
) inherits filebeat::params {

$kernel_fail_message = "${::kernel} is not supported by filebeat."

validate_bool($manage_repo, $prospectors_merge)

if $major_version == undef and $::filebeat_version == undef {
$real_version = '5'
} elsif $major_version == undef and versioncmp($::filebeat_version, '5.0.0') >= 0 {
$real_version = '5'
} elsif $major_version == undef and versioncmp($::filebeat_version, '5.0.0') < 0 {
$real_version = '1'
} else {
$real_version = $major_version
}

if $conf_template != undef {
$real_conf_template = $conf_template
} elsif $real_version == '1' {
if versioncmp('1.9.1', $::rubyversion) > 0 {
$real_conf_template = "${module_name}/filebeat.yml.ruby18.erb"
} else {
$real_conf_template = "${module_name}/filebeat.yml.erb"
}
} elsif $real_version == '5' {
if $use_generic_template {
$real_conf_template = "${module_name}/filebeat.yml.erb"
} else {
$real_conf_template = "${module_name}/filebeat5.yml.erb"
}
}


if $prospectors_merge {
$prospectors_final = hiera_hash('filebeat::prospectors', $prospectors)
} else {
Expand Down
23 changes: 16 additions & 7 deletions manifests/params.pp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,21 +6,30 @@
$spool_size = 2048
$idle_timeout = '5s'
$publish_async = false
$shutdown_timeout = 0
$beat_name = $::fqdn
$tags = []
$queue_size = 1000
$max_procs = undef
$registry_file = '.filebeat'
$config_dir_mode = '0755'
$config_file_mode = '0644'
$purge_conf_dir = true
$fields = {}
$fields_under_root = false
$outputs = {}
$shipper = {}
$logging = {}
$run_options = {}
$use_generic_template = false

if versioncmp('1.9.1', $::rubyversion) > 0 {
$conf_template = "${module_name}/filebeat.yml.ruby18.erb"
} else {
$conf_template = "${module_name}/filebeat.yml.erb"
}

# These are irrelevant as long as the template is set based on the major_version parameter
# if versioncmp('1.9.1', $::rubyversion) > 0 {
# $conf_template = "${module_name}/filebeat.yml.ruby18.erb"
# } else {
# $conf_template = "${module_name}/filebeat.yml.erb"
# }
#
case $::kernel {
'Linux' : {
$config_file = '/etc/filebeat/filebeat.yml'
Expand All @@ -43,7 +52,7 @@
'Windows' : {
$config_file = 'C:/Program Files/Filebeat/filebeat.yml'
$config_dir = 'C:/Program Files/Filebeat/conf.d'
$download_url = 'https://download.elastic.co/beats/filebeat/filebeat-1.2.3-windows.zip'
$download_url = 'https://download.elastic.co/beats/filebeat/filebeat-1.3.1-windows.zip'
$install_dir = 'C:/Program Files'
$tmp_dir = 'C:/Windows/Temp'
$service_provider = undef
Expand Down
Loading