Skip to content
Puppet module to manage Logstash
Branch: master
Clone or download
Latest commit 4f06ff1 Dec 4, 2018
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
Vagrantfile.d Fix docs for environment param on plugin type Dec 5, 2018
files Remove file_concat logic Dec 7, 2016
manifests Fix docs for environment param on plugin type Dec 5, 2018
spec Test restart_on_change works for startup options Nov 13, 2018
templates No path.config for centralized pipeline management Apr 2, 2018
.fixtures.yml Fix resource defaults for plugins on Puppet 5 #354 Dec 6, 2017
.gitignore Fix tests for beta versions Sep 12, 2017
.pmtignore update common files Jan 13, 2016
.projectile Begin move to puppet-strings. Remove params class. Dec 13, 2016
.rubocop.yml Add rubocop control file May 3, 2017
.travis.yml Don't test LS6 on Debian 7, it's not supported Feb 13, 2018
CHANGELOG Bump version to 6.1.5 Nov 13, 2018
CONTRIBUTING.md Run Beaker against Debian 7 on Travis. Apr 17, 2016
CONTRIBUTORS
Gemfile Add "rgen" gem Dec 6, 2017
LICENSE update common files Jan 13, 2016
Makefile Remove hopelessly obsolete unit test suite Mar 21, 2017
README.md Fix docs for environment param on plugin type Dec 5, 2018
Rakefile Fix "rake lint" output May 30, 2018
Vagrantfile Refactor Vagrantfile Nov 14, 2017
metadata.json Bump version to 6.1.5 Nov 13, 2018

README.md

elastic/logstash

A Puppet module for managing and configuring Logstash.

Build Status

Logstash Versions

This module, "elastic/logstash" supports only Logstash 5.x and 6.x. For earlier Logstash versions, support is provided by the legacy module "elasticsearch/logstash".

Requirements

  • Puppet 4.6.1 or better.
  • The stdlib module.
  • Logstash itself requires Java 8. The "puppetlabs/java" module is recommended for installing Java. This module will not install Java.

Optional:

  • The elastic_stack module when using automatic repository management.
  • The apt (>= 2.0.0) module when using repo management on Debian/Ubuntu.
  • The zypprepo module when using repo management on SLES/SuSE.

Quick Start

This minimum viable configuration ensures that Logstash is installed, enabled, and running:

include logstash

# You must provide a valid pipeline configuration for the service to start.
logstash::configfile { 'my_ls_config':
  content => template('path/to/config.file'),
}

Package and service options

Choosing a Logstash minor version

class { 'logstash':
  version => '6.0.0',
}

Choosing a Logstash major version

This module uses the related "elastic/elastic_stack" module to manage package repositories. Since there is a separate repository for each major version of the Elastic stack, if you don't want the default version (6), it's necessary to select which version to configure, like this:

class { 'elastic_stack::repo':
  version => 5,
}

class { 'logstash':
  version => '5.6.4',
}

Manual repository management

You may want to manage repositories manually. You can disable automatic repository management like this:

class { 'logstash':
  manage_repo => false,
}

Using an explicit package source

Rather than use your distribution's repository system, you can specify an explicit package to fetch and install.

From an HTTP/HTTPS/FTP URL

class { 'logstash':
  package_url => 'https://artifacts.elastic.co/downloads/logstash/logstash-5.1.1.rpm',
}

From a 'puppet://' URL

class { 'logstash':
  package_url => 'puppet:///modules/my_module/logstash-5.1.1.rpm',
}

From a local file on the agent

class { 'logstash':
  package_url => 'file:///tmp/logstash-5.1.1.rpm',
}

Allow automatic point-release upgrades

class { 'logstash':
  auto_upgrade => true,
}

Do not run as a service

class { 'logstash':
  status => 'disabled',
}

Disable automatic restarts

Under normal circumstances, changing a configuration will trigger a restart of the service. This behaviour can be disabled:

class { 'logstash':
  restart_on_change => false,
}

Disable and remove Logstash

class { 'logstash':
  ensure => 'absent',
}

Logstash config files

Settings

Logstash uses several files to define settings for the service and associated Java runtime. The settings files can be configured with class parameters.

logstash.yml with flat keys

class { 'logstash':
  settings => {
    'pipeline.batch.size'  => 25,
    'pipeline.batch.delay' => 5,
  }
}

logstash.yml with nested keys

class { 'logstash':
  settings => {
    'pipeline' => {
      'batch' => {
        'size'  => 25,
        'delay' => 5,
      }
    }
  }
}

jvm.options

class { 'logstash':
  jvm_options => [
    '-Xms1g',
    '-Xmx1g',
  ]
}

startup.options

class { 'logstash':
  startup_options => {
    'LS_NICE' => '10',
  }
}

pipelines.yml

class { 'logstash':
  pipelines => [
    {
      "pipeline.id" => "pipeline_one",
      "path.config" =>  "/usr/local/etc/logstash/pipeline-1/one.conf",
    },
    {
      "pipeline.id" => "pipeline_two",
      "path.config" =>  "/usr/local/etc/logstash/pipeline-2/two.conf",
    }
  ]
}

Note that specifying pipelines will automatically remove the default path.config setting from logstash.yml, since this is incompatible with pipelines.yml.

Enabling centralized pipeline management with xpack.management.enabled will also remove the default path.config.

Pipeline Configuration

Pipeline configuration files can be declared with the logstash::configfile type.

logstash::configfile { 'inputs':
  content => template('path/to/input.conf.erb'),
}

or

logstash::configfile { 'filters':
  source => 'puppet:///path/to/filter.conf',
}

For simple cases, it's possible to provide your Logstash config as an inline string:

logstash::configfile { 'basic_ls_config':
  content => 'input { heartbeat {} } output { null {} }',
}

You can also specify the exact path for the config file, which is particularly useful with multiple pipelines:

logstash::configfile { 'config_for_pipeline_two':
  content => 'input { heartbeat {} } output { null {} }',
  path    => '/usr/local/etc/logstash/pipeline-2/two.conf',
}

If you want to use Hiera to specify your configs, include the following create_resources call in your manifest:

create_resources('logstash::configfile', hiera('my_logstash_configs'))

...and then create a data structure like this in Hiera:

---
my_logstash_configs:
  nginx:
    template: site_logstash/nginx.conf.erb
  syslog:
    template: site_logstash/syslog.conf.erb

In this example, templates for the config files are stored in the custom, site-specific module "site_logstash".

Patterns

Many plugins (notably Grok) use patterns. While many are included in Logstash already, additional site-specific patterns can be managed as well.

logstash::patternfile { 'extra_patterns':
  source => 'puppet:///path/to/extra_pattern',
}

By default the resulting filename of the pattern will match that of the source. This can be over-ridden:

logstash::patternfile { 'extra_patterns_firewall':
  source   => 'puppet:///path/to/extra_patterns_firewall_v1',
  filename => 'extra_patterns_firewall',
}

IMPORTANT NOTE: Using logstash::patternfile places new patterns in the correct directory, however, it does NOT cause the path to be included automatically for filters (example: grok filter). You will still need to include this path (by default, /etc/logstash/patterns/) explicitly in your configurations.

Example: If using 'grok' in one of your configurations, you must include the pattern path in each filter like this:

# Note: this example is Logstash configuration, not a Puppet resource.
# Logstash and Puppet look very similar!
grok {
  patterns_dir => "/etc/logstash/patterns/"
  ...
}

Plugin management

Installing by name (from RubyGems.org)

logstash::plugin { 'logstash-input-beats': }

Installing from a local Gem

logstash::plugin { 'logstash-input-custom':
  source => '/tmp/logstash-input-custom-0.1.0.gem',
}

Installing from a 'puppet://' URL

logstash::plugin { 'logstash-filter-custom':
  source => 'puppet:///modules/my_ls_module/logstash-filter-custom-0.1.0.gem',
}

Installing from an 'http(s)://' URL

logstash::plugin { 'x-pack':
  source => 'https://artifacts.elastic.co/downloads/packs/x-pack/x-pack-5.3.0.zip',
}

Controling the environment for the logstash-plugin command

logstash::plugin { 'logstash-input-websocket':
  environment => 'LS_JAVA_OPTS=-Xms1g -Xmx1g',
}

Support

Need help? Join us in #logstash on Freenode IRC or on the https://discuss.elastic.co/c/logstash discussion forum.

You can’t perform that action at this time.