6 changes: 4 additions & 2 deletions .fixtures.yml
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
fixtures:
repositories:
"stdlib": "git://github.com/puppetlabs/puppetlabs-stdlib.git"
"apt": "git://github.com/puppetlabs/puppetlabs-apt.git"
"stdlib": "https://github.com/puppetlabs/puppetlabs-stdlib.git"
"apt": "https://github.com/puppetlabs/puppetlabs-apt.git"
"powershell": "https://github.com/puppetlabs/puppetlabs-powershell.git"
"remotefile": "https://github.com/lwf/puppet-remote_file.git"
symlinks:
"filebeat": "#{source_dir}"
19 changes: 19 additions & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
Changelog
=========

## Unreleased
[Full Changelog](https://github.com/pcfens/puppet-filebeat/compare/v0.4.0...HEAD)

## [v0.4.0](https://github.com/pcfens/puppet-filebeat/tree/v0.4.0)
[Full Changelog](https://github.com/pcfens/puppet-filebeat/compare/v0.3.1...v0.4.0)

This is the first release that includes changelog. Since v0.3.1:

**Fixed Bugs**
- 'fields' parse error in prospector.yml template [\#7](https://github.com/pcfens/puppet-filebeat/pull/7)

**New Features**
- Windows support [\#3](https://github.com/pcfens/puppet-filebeat/pull/3)
- Requires the [`puppetlabs/powershell`](https://forge.puppetlabs.com/puppetlabs/powershell)
and [`lwf/remote_file`](https://forge.puppetlabs.com/lwf/remote_file) modules.
- Config file and folder permissions can be managed [\#8](https://github.com/pcfens/puppet-filebeat/pull/8)
115 changes: 110 additions & 5 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,16 @@

1. [Description](#description)
2. [Setup - The basics of getting started with filebeat](#setup)
* [What filebeat affects](#what-filebeat-affects)
* [Setup requirements](#setup-requirements)
* [Beginning with filebeat](#beginning-with-filebeat)
- [What filebeat affects](#what-filebeat-affects)
- [Setup requirements](#setup-requirements)
- [Beginning with filebeat](#beginning-with-filebeat)
3. [Usage - Configuration options and additional functionality](#usage)
4. [Limitations - OS compatibility, etc.](#limitations)
5. [Development - Guide for contributing to the module](#development)
4. [Reference](#reference)
- [Public Classes](#public-classes)
- [Private Classes](#private-classes)
- [Public Defines](#public-defines)
5. [Limitations - OS compatibility, etc.](#limitations)
6. [Development - Guide for contributing to the module](#development)

## Description

Expand Down Expand Up @@ -96,6 +100,107 @@ filebeat::prospector { 'syslogs':
}
```

## Reference
- [**Public Classes**](#public-classes)
- [Class: filebeat](#class-filebeat)
- [**Private Classes**](#private-classes)
- [Class: filebeat::config](#class-filebeatconfig)
- [Class: filebeat::install](#class-filebeatinstall)
- [Class: filebeat::params](#class-filebeatparams)
- [Class: filebeat::repo](#class-filebeatrepo)
- [Class: filebeat::service](#class-filebeatservice)
- [Class: filebeat::install::linux](#class-filebeatinstalllinux)
- [Class: filebeat::install::windows](#class-filebeatinstallwindows)
- [**Public Defines**](#public-defines)
- [Define: filebeat::prospector](#define-filebeatprospector)

### Public Classes

#### Class: `filebeat`

Installs and configures filebeat.

**Parameters within `filebeat`**
- `package_ensure`: [String] The ensure parameter for the filebeat package (default: present)
- `manage_repo`: [Boolean] Whether or not the upstream (elastic) repo should be configured or not (default: true)
- `service_ensure`: [String] The ensure parameter on the filebeat service (default: running)
- `service_enable`: [String] The enable parameter on the filebeat service (default: true)
- `spool_size`: [Integer] How large the spool should grow before being flushed to the network (default: 1024)
- `idle_timeout`: [String] How often the spooler should be flushed even if spool size isn't reached (default: 5s)
- `registry_file`: [String] The registry file used to store positions, absolute or relative to working directory (default .filebeat)
- `config_dir`: [String] The directory where prospectors should be defined (default: /etc/filebeat/conf.d)
- `config_dir_mode`: [String] The permissions mode set on the configuration directory (default: 0755)
- `config_file_mode`: [String] The permissions mode set on configuration files (default: 0644)
- `purge_conf_dir`: [Boolean] Should files in the prospector configuration directory not managed by puppet be automatically purged
- `outputs`: [Hash] Will be converted to YAML for the required outputs section of the configuration (see documentation, and above)
- `shipper`: [Hash] Will be converted to YAML to create the optional shipper section of the filebeat config (see documentation)
- `logging`: [Hash] Will be converted to YAML to create the optional logging section of the filebeat config (see documentation)
- `conf_template`: [String] The configuration template to use to generate the main filebeat.yml config file
- `download_url`: [String] The URL of the zip file that should be downloaded to install filebeat (windows only)
- `install_dir`: [String] Where filebeat should be installed (windows only)
- `tmp_dir`: [String] Where filebeat should be temporarily downloaded to so it can be installed (windows only)
- `prospectors`: [Hash] Prospectors that will be created. Commonly used to create prospectors using hiera

### Private Classes

#### Class: `filebeat::config`

Creates the configuration files required for filebeat (but not the prospectors)

#### Class: `filebeat::install`

Calls the correct installer class based on the kernel fact.

#### Class: `filebeat::params`

Sets default parameters for `filebeat` based on the OS and other facts.

#### Class: `filebeat::repo`

Installs the yum or apt repository for the system package manager to install filebeat.

#### Class: `filebeat::service`

Configures and manages the filebeat service.

#### Class: `filebeat::install::linux`

Install the filebeat package on Linux kernels.

#### Class: `filebeat::install::windows`

Downloads, extracts, and installs the filebeat zip file in Windows.

### Public Defines

#### Define: `filebeat::prospector`

Installs a configuration file for a prospector.

Be sure to read the [filebeat configuration details](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-configuration-details.html)
to fully understand what these parameters do.

**Parameters for `filebeat::prospector`**
- `ensure`: The ensure parameter on the prospector configuration file. (default: present)
- `paths`: [Array] The paths, or blobs that should be handled by the prospector. (required)
- `encoding`: [String] The file encoding. (default: plain)
- `input_type`: [String] log or stdin - where filebeat reads the log from (default:log)
- `fields`: [Hash] Optional fields to add information to the output (default: {})
- `fields_under_root`: [Boolean] Should the `fields` parameter fields be stored at the top level of indexed documents.
- `ignore_older`: [String] Files older than this field will be ignored by filebeat (default: 24h)
- `log_type`: [String] The type parameter to send to logstash (optional - default: log)
- `scan_frequency`: [String] How often should the prospector check for new files (default: 10s)
- `harvester_buffer_size`: [Integer] The buffer size the harvester uses when fetching the file (default: 16384)
- `tail_files`: [Boolean] If true, filebeat starts reading new files at the end instead of the beginning (default: false)
- `backoff`: [String] How long filebeat should wait between scanning a file after reaching EOF (default: 1s)
- `max_backoff`: [String] The maximum wait time to scan a file for new lines to ship (default: 10s)
- `backoff_factor`: [Integer] `backoff` is multiplied by this parameter until `max_backoff` is reached to
determine the actual backoff (default: 2)
- `partial_line_waiting`: [String] How long should the prospector wait before shipping a file with
a potentially incomplete last line (default: 5s)
- `force_close_files`: [Boolean] Should filebeat forcibly close a file when renamed (default: false)


## Limitations

This module doesn't load the [elasticsearch index template](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-getting-started.html#filebeat-template) into elasticsearch (required when shipping
Expand Down
68 changes: 46 additions & 22 deletions manifests/config.pp
View file
Original file line number Diff line number Diff line change
@@ -1,34 +1,58 @@
class filebeat::config {

$filebeat_config = {
'filebeat' => {
'filebeat' => {
'spool_size' => $filebeat::spool_size,
'idle_timeout' => $filebeat::idle_timeout,
'registry_file' => $filebeat::registry_file,
'config_dir' => $filebeat::config_dir,
},
'output' => $filebeat::outputs,
'shipper' => $filebeat::shipper,
'logging' => $filebeat::logging,
'output' => $filebeat::outputs,
'shipper' => $filebeat::shipper,
'logging' => $filebeat::logging,
'runoptions' => $filebeat::run_options,
}

file {'filebeat.yml':
ensure => file,
path => '/etc/filebeat/filebeat.yml',
content => template("${module_name}/filebeat.yml.erb"),
owner => 'root',
group => 'root',
mode => '0644',
notify => Service['filebeat'],
}
case $::kernel {
'Linux' : {
file {'filebeat.yml':
ensure => file,
path => '/etc/filebeat/filebeat.yml',
content => template("${module_name}/filebeat.yml.erb"),
owner => 'root',
group => 'root',
mode => $filebeat::config_file_mode,
notify => Service['filebeat'],
}

file {'filebeat-config-dir':
ensure => directory,
path => $filebeat::config_dir,
owner => 'root',
group => 'root',
mode => $filebeat::config_dir_mode,
recurse => $filebeat::purge_conf_dir,
purge => $filebeat::purge_conf_dir,
}
} # end Linux

'Windows' : {
file {'filebeat.yml':
ensure => file,
path => 'C:/Program Files/Filebeat/filebeat.yml',
content => template("${module_name}/filebeat.yml.erb"),
notify => Service['filebeat'],
}

file {'filebeat-config-dir':
ensure => directory,
path => $filebeat::config_dir,
recurse => $filebeat::purge_conf_dir,
purge => $filebeat::purge_conf_dir,
}
} # end Windows

file {'filebeat-config-dir':
ensure => directory,
path => $filebeat::config_dir,
owner => 'root',
group => 'root',
mode => '0755',
recurse => $filebeat::purge_conf_dir,
purge => $filebeat::purge_conf_dir,
default : {
fail($filebeat::kernel_fail_message)
}
}
}
54 changes: 30 additions & 24 deletions manifests/init.pp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,28 +20,42 @@
# @param idle_timeout [String] How often the spooler should be flushed even if spool size isn't reached (default: 5s)
# @param registry_file [String] The registry file used to store positions, absolute or relative to working directory (default .filebeat)
# @param config_dir [String] The directory where prospectors should be defined (default: /etc/filebeat/conf.d)
# @param config_dir_mode [String] The unix permissions mode set on the configuration directory (default: 0755)
# @param config_file_mode [String] The unix permissions mode set on configuration files (default: 0644)
# @param purge_conf_dir [Boolean] Should files in the prospector configuration directory not managed by puppet be automatically purged
# @param outputs [Hash] Will be converted to YAML for the required outputs section of the configuration (see documentation, and above)
# @param shipper [Hash] Will be converted to YAML to create the optional shipper section of the filebeat config (see documentation)
# @param logging [Hash] Will be converted to YAML to create the optional logging section of the filebeat config (see documentation)
# @param conf_template [String] The configuration template to use to generate the main filebeat.yml config file
# @param download_url [String] The URL of the zip file that should be downloaded to install filebeat (windows only)
# @param install_dir [String] Where filebeat should be installed (windows only)
# @param tmp_dir [String] Where filebeat should be temporarily downloaded to so it can be installed (windows only)
# @param prospectors [Hash] Prospectors that will be created. Commonly used to create prospectors using hiera
class filebeat (
$package_ensure = $filebeat::params::package_ensure,
$manage_repo = $filebeat::params::manage_repo,
$service_ensure = $filebeat::params::service_ensure,
$service_enable = $filebeat::params::service_enable,
$spool_size = $filebeat::params::spool_size,
$idle_timeout = $filebeat::params::idle_timeout,
$registry_file = $filebeat::params::registry_file,
$config_dir = $filebeat::params::config_dir,
$purge_conf_dir = $filebeat::params::purge_conf_dir,
$outputs = $filebeat::params::outputs,
$shipper = $filebeat::params::shipper,
$logging = $filebeat::params::logging,
$conf_template = $filebeat::params::conf_template,
$prospectors = {},
$package_ensure = $filebeat::params::package_ensure,
$manage_repo = $filebeat::params::manage_repo,
$service_ensure = $filebeat::params::service_ensure,
$service_enable = $filebeat::params::service_enable,
$spool_size = $filebeat::params::spool_size,
$idle_timeout = $filebeat::params::idle_timeout,
$registry_file = $filebeat::params::registry_file,
$config_dir = $filebeat::params::config_dir,
$config_dir_mode = $filebeat::params::config_dir_mode,
$config_file_mode = $filebeat::params::config_file_mode,
$purge_conf_dir = $filebeat::params::purge_conf_dir,
$outputs = $filebeat::params::outputs,
$shipper = $filebeat::params::shipper,
$logging = $filebeat::params::logging,
$run_options = $filebeat::params::run_options,
$conf_template = $filebeat::params::conf_template,
$download_url = $filebeat::params::download_url,
$install_dir = $filebeat::params::install_dir,
$tmp_dir = $filebeat::params::tmp_dir,
$prospectors = {},
) inherits filebeat::params {

$kernel_fail_message = "${::kernel} is not supported by filebeat."

validate_bool($manage_repo)
validate_hash($outputs, $logging, $prospectors)
validate_string($idle_timeout, $registry_file, $config_dir, $package_ensure)
Expand All @@ -51,18 +65,10 @@
}

anchor { 'filebeat::begin': } ->
class { 'filebeat::package': } ->
class { 'filebeat::install': } ->
class { 'filebeat::config': } ->
class { 'filebeat::service': } ->
anchor { 'filebeat::end':}

if $manage_repo {
include filebeat::repo

Anchor['filebeat::begin'] ->
Class['filebeat::repo'] ->
Class['filebeat::package']
}
anchor { 'filebeat::end': }

if !empty($prospectors) {
create_resources('filebeat::prospector', $prospectors)
Expand Down
17 changes: 17 additions & 0 deletions manifests/install.pp
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
class filebeat::install {
case $::kernel {
'Linux': {
contain filebeat::install::linux
if $::filebeat::manage_repo {
contain filebeat::repo
Class['filebeat::repo'] -> Class['filebeat::install::linux']
}
}
'Windows': {
contain filebeat::install::windows
}
default: {
fail($filebeat::kernel_fail_message)
}
}
}
2 changes: 1 addition & 1 deletion manifests/package.pp → manifests/install/linux.pp
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
class filebeat::package {
class filebeat::install::linux {
package {'filebeat':
ensure => $filebeat::package_ensure,
}
Expand Down
44 changes: 44 additions & 0 deletions manifests/install/windows.pp
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
class filebeat::install::windows {
$filename = regsubst($filebeat::download_url, '^https.*\/([^\/]+)\.[^.].*', '\1')
$foldername = 'Filebeat'

file { $filebeat::tmp_dir:
ensure => directory
}

file { $filebeat::install_dir:
ensure => directory
}

remote_file {"${filebeat::tmp_dir}/${filename}.zip":
ensure => present,
source => $filebeat::download_url,
require => File[$filebeat::tmp_dir],
verify_peer => false,
}

exec { "unzip ${filename}":
command => "\$sh=New-Object -COM Shell.Application;\$sh.namespace((Convert-Path '${filebeat::install_dir}')).Copyhere(\$sh.namespace((Convert-Path '${filebeat::tmp_dir}/${filename}.zip')).items(), 16)",
creates => "${filebeat::install_dir}/Filebeat",
provider => powershell,
require => [
File[$filebeat::install_dir],
Remote_file["${filebeat::tmp_dir}/${filename}.zip"],
],
}

exec { 'rename folder':
command => "Rename-Item '${filebeat::install_dir}/${filename}' Filebeat",
creates => "${filebeat::install_dir}/Filebeat",
provider => powershell,
require => Exec["unzip ${filename}"],
}

exec { "install ${filename}":
cwd => "${filebeat::install_dir}/Filebeat",
command => './install-service-filebeat.ps1',
onlyif => 'if(Get-WmiObject -Class Win32_Service -Filter "Name=\'filebeat\'") { exit 1 } else {exit 0 }',
provider => powershell,
require => Exec['rename folder'],
}
}
Loading