Showing with 180 additions and 32 deletions.
  1. +16 −1 CHANGELOG.md
  2. +17 −1 README.md
  3. +17 −1 manifests/prospector.pp
  4. +1 −1 metadata.json
  5. +69 −7 spec/defines/prospector_spec.rb
  6. +60 −21 templates/prospector.yml.erb
17 changes: 16 additions & 1 deletion CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,22 @@ Changelog
=========

## Unreleased
[Full Changelog](https://github.com/pcfens/puppet-filebeat/compare/v0.4.1...HEAD)
[Full Changelog](https://github.com/pcfens/puppet-filebeat/compare/v0.5.0...HEAD)


## [v0.5.0](https://github.com/pcfens/puppet-filebeat/tree/v0.5.0)
[Full Changelog](https://github.com/pcfens/puppet-filebeat/compare/v0.4.1...v0.5.0)

- For prospectors, deprecate `log_type` in favor of `doc_type` to better
match the actual configuration parameter. `document_type` is not used because
it causes errors when running with a puppet master. `log_type` will be fully
removed before module version 1.0.
[\#9](https://github.com/pcfens/puppet-filebeat/issues/9)

**New Features**
- Add support for `exclude_files`, `exclude_lines`, `include_lines`, and `multiline`.
Use of the new parameters requires a filebeat version >= 1.1
([\#10](https://github.com/pcfens/puppet-filebeat/issues/10), [\#11](https://github.com/pcfens/puppet-filebeat/issues/11))

## [v0.4.1](https://github.com/pcfens/puppet-filebeat/tree/v0.4.1)
[Full Changelog](https://github.com/pcfens/puppet-filebeat/compare/v0.4.0...v0.4.1)
Expand Down
18 changes: 17 additions & 1 deletion README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,12 @@ filebeat::prospector { 'syslogs':
}
```

#### Multiline Logs

Filebeat prospectors (versions >= 1.1) can handle multiline log entries. The `multiline`
parameter accepts a hash containing `pattern`, `negate`, `match`, `max_lines`, and `timeout`
as documented in the filebeat [configuration documentation](https://www.elastic.co/guide/en/beats/filebeat/1.1/filebeat-configuration-details.html#multiline).

## Reference
- [**Public Classes**](#public-classes)
- [Class: filebeat](#class-filebeat)
Expand Down Expand Up @@ -183,12 +189,15 @@ to fully understand what these parameters do.
**Parameters for `filebeat::prospector`**
- `ensure`: The ensure parameter on the prospector configuration file. (default: present)
- `paths`: [Array] The paths, or blobs that should be handled by the prospector. (required)
- `exclude_files`: [Array] Files that match any regex in the list are excluded from filebeat (default: [])
- `encoding`: [String] The file encoding. (default: plain)
- `input_type`: [String] log or stdin - where filebeat reads the log from (default:log)
- `fields`: [Hash] Optional fields to add information to the output (default: {})
- `fields_under_root`: [Boolean] Should the `fields` parameter fields be stored at the top level of indexed documents.
- `ignore_older`: [String] Files older than this field will be ignored by filebeat (default: 24h)
- `log_type`: [String] The type parameter to send to logstash (optional - default: log)
- `log_type`: [String] (Deprecated - use `doc_type`) The document_type setting (optional - default: log)
- `doc_type`: [String] The event type to used for published lines, used as type field in logstash
and elasticsearch (optional - default: log)
- `scan_frequency`: [String] How often should the prospector check for new files (default: 10s)
- `harvester_buffer_size`: [Integer] The buffer size the harvester uses when fetching the file (default: 16384)
- `tail_files`: [Boolean] If true, filebeat starts reading new files at the end instead of the beginning (default: false)
Expand All @@ -199,6 +208,13 @@ to fully understand what these parameters do.
- `partial_line_waiting`: [String] How long should the prospector wait before shipping a file with
a potentially incomplete last line (default: 5s)
- `force_close_files`: [Boolean] Should filebeat forcibly close a file when renamed (default: false)
- `include_lines`: [Array] A list of regular expressions to match the lines that you want to include.
Ignored if empty (default: [])
- `exclude_lines`: [Array] A list of regular expressions to match the files that you want to exclude.
Ignored if empty (default: [])
- `max_bytes`: [Integer] The maximum number of bytes that a single log message can have (default: 10485760)
- `multiline`: [Hash] Options that control how Filebeat handles log messages that span multiple lines.
[See above](#multiline-logs). (default: {})


## Limitations
Expand Down
18 changes: 17 additions & 1 deletion manifests/prospector.pp
View file
Original file line number Diff line number Diff line change
@@ -1,12 +1,14 @@
define filebeat::prospector (
$ensure = present,
$paths = [],
$exclude_files = [],
$encoding = 'plain',
$input_type = 'log',
$fields = {},
$fields_under_root = false,
$ignore_older = '24h',
$log_type = 'log',
$log_type = undef,
$doc_type = 'log',
$scan_frequency = '10s',
$harvester_buffer_size = 16384,
$tail_files = false,
Expand All @@ -15,8 +17,22 @@
$backoff_factor = 2,
$partial_line_waiting = '5s',
$force_close_files = false,
$include_lines = [],
$exclude_lines = [],
$max_bytes = '10485760',
$multiline = {},
) {

validate_hash($fields, $multiline)
validate_array($paths, $exclude_files, $include_lines, $exclude_lines)

if $log_type {
warning('log_type is deprecated, and will be removed prior to a v1.0 release so parameters match the filebeat documentation - use doc_type instead')
$real_doc_type = $log_type
} else {
$real_doc_type = $doc_type
}

case $::kernel {
'Linux' : {
file { "filebeat-${name}":
Expand Down
2 changes: 1 addition & 1 deletion metadata.json
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "pcfens-filebeat",
"version": "0.4.1",
"version": "0.5.0",
"author": "pcfens",
"license": "Apache-2.0",
"summary": "A module to install and manage the filebeat log shipper",
Expand Down
76 changes: 69 additions & 7 deletions spec/defines/prospector_spec.rb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
}'
end
let :title do
'apache-logs'
'test-logs'
end

context 'with no parameters' do
Expand All @@ -33,12 +33,12 @@
:paths => [
'/var/log/apache2/*.log',
],
:log_type => 'apache',
:doc_type => 'apache',
}
end

it { is_expected.to contain_file('filebeat-apache-logs').with(
:path => '/etc/filebeat/conf.d/apache-logs.yml',
it { is_expected.to contain_file('filebeat-test-logs').with(
:path => '/etc/filebeat/conf.d/test-logs.yml',
:mode => '0644',
:content => 'filebeat:
prospectors:
Expand All @@ -57,6 +57,67 @@
max_backoff: 10s
backoff_factor: 2
partial_line_waiting: 5s
max_bytes: 10485760
',
)}
end
context 'with some java like multiline settings' do
let :params do
{
:paths => [
'/var/log/java_app/some.log',
],
:doc_type => 'java_app',
:exclude_lines => [
'^DEBUG',
],
:include_lines => [
'^ERROR',
'^WARN',
],
:exclude_files => [
'.gz$',
],
:multiline => {
'pattern' => '^\[',
'negate' => 'true',
'match' => 'after',
},
}
end

it { is_expected.to contain_file('filebeat-test-logs').with(
:path => '/etc/filebeat/conf.d/test-logs.yml',
:mode => '0644',
:content => 'filebeat:
prospectors:
- paths:
- /var/log/java_app/some.log
exclude_files:
- .gz$
encoding: plain
fields_under_root: false
input_type: log
ignore_older: 24h
document_type: java_app
scan_frequency: 10s
harvester_buffer_size: 16384
tail_files: false
force_close_files: false
backoff: 1s
max_backoff: 10s
backoff_factor: 2
partial_line_waiting: 5s
max_bytes: 10485760
multiline:
pattern: ^\[
negate: true
match: after
include_lines:
- ^ERROR
- ^WARN
exclude_lines:
- ^DEBUG
',
)}
end
Expand All @@ -74,12 +135,12 @@
:paths => [
'C:/Program Files/Apache Software Foundation/Apache2.2/*.log',
],
:log_type => 'apache',
:doc_type => 'apache',
}
end

it { is_expected.to contain_file('filebeat-apache-logs').with(
:path => 'C:/Program Files/Filebeat/conf.d/apache-logs.yml',
it { is_expected.to contain_file('filebeat-test-logs').with(
:path => 'C:/Program Files/Filebeat/conf.d/test-logs.yml',
:content => 'filebeat:
prospectors:
- paths:
Expand All @@ -97,6 +158,7 @@
max_backoff: 10s
backoff_factor: 2
partial_line_waiting: 5s
max_bytes: 10485760
',
)}
end
Expand Down
81 changes: 60 additions & 21 deletions templates/prospector.yml.erb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,15 @@ filebeat:
<%- @paths.each do |log_path| -%>
- <%= log_path %>
<%- end -%>
<%- if @exclude_files.length > 0 -%>
exclude_files:
<%- @exclude_files.each do |exclude_file| -%>
- <%= exclude_file %>
<%- end -%>
<%- end -%>
<%- if @encoding -%>
encoding: <%= @encoding -%>
<%- end %>
encoding: <%= @encoding %>
<%- end -%>
<%- if @fields.length > 0 -%>
fields:
<%- @fields.each_pair do |k, v| -%>
Expand All @@ -15,31 +21,64 @@ filebeat:
<%- end -%>
fields_under_root: <%= @fields_under_root %>
<%- if @input_type -%>
input_type: <%= @input_type -%>
<%- end %>
input_type: <%= @input_type %>
<%- end -%>
<%- if @ignore_older -%>
ignore_older: <%= @ignore_older -%>
<%- end %>
<%- if @log_type -%>
document_type: <%= @log_type -%>
<%- end %>
ignore_older: <%= @ignore_older %>
<%- end -%>
<%- if @real_doc_type -%>
document_type: <%= @real_doc_type %>
<%- end -%>
<%- if @scan_frequency -%>
scan_frequency: <%= @scan_frequency -%>
<%- end %>
scan_frequency: <%= @scan_frequency %>
<%- end -%>
<%- if @harvester_buffer_size -%>
harvester_buffer_size: <%= @harvester_buffer_size -%>
<%- end %>
harvester_buffer_size: <%= @harvester_buffer_size %>
<%- end -%>
tail_files: <%= @tail_files %>
force_close_files: <%= @force_close_files %>
<%- if @backoff -%>
backoff: <%= @backoff -%>
<%- end %>
backoff: <%= @backoff %>
<%- end -%>
<%- if @max_backoff -%>
max_backoff: <%= @max_backoff -%>
<%- end %>
max_backoff: <%= @max_backoff %>
<%- end -%>
<%- if @backoff_factor -%>
backoff_factor: <%= @backoff_factor -%>
<%- end %>
backoff_factor: <%= @backoff_factor %>
<%- end -%>
<%- if @partial_line_waiting -%>
partial_line_waiting: <%= @partial_line_waiting -%>
<%- end %>
partial_line_waiting: <%= @partial_line_waiting %>
<%- end -%>
<%- if @max_bytes -%>
max_bytes: <%= @max_bytes %>
<%- end -%>
<%- if @multiline.length > 0 -%>
multiline:
<%- if @multiline['pattern'] -%>
pattern: <%= @multiline['pattern'] %>
<%- end -%>
<%- if @multiline['negate'] -%>
negate: <%= @multiline['negate'] %>
<%- end -%>
<%- if @multiline['match'] -%>
match: <%= @multiline['match'] %>
<%- end -%>
<%- if @multiline['max_lines'] -%>
max_lines: <%= @multiline['max_lines'] %>
<%- end -%>
<%- if @multiline['timeout'] -%>
timeout: <%= @multiline['timeout'] %>
<%- end -%>
<%- end -%>
<%- if @include_lines.length > 0 -%>
include_lines:
<%- @include_lines.each do |include_line| -%>
- <%= include_line %>
<%- end -%>
<%- end -%>
<%- if @exclude_lines.length > 0 -%>
exclude_lines:
<%- @exclude_lines.each do |exclude_line| -%>
- <%= exclude_line %>
<%- end -%>
<%- end -%>