Puppet Pulse - Splunk app for Puppet
Python CSS Shell
Latest commit 7d8a650 Jun 3, 2015 @wcooley Convert run stats input to number
Without the `tonumber()` function, the `inputlookup` result comes in as
a string, which compares lexigraphically.

Also, the leading pipe "|" is not needed in the subsearch, so remove it.

Also, use the `fields` command, not `table`, to extract just the query
field.
Permalink
Failed to load latest commit information.
appserver/static Update app logo and CSS for new bluish background Apr 20, 2013
bin
default
local
lookups Ignore everything under lookups by default Jul 1, 2012
metadata
misc
.gitignore
LICENSE Add copyright notice & license info Jan 31, 2013
README.md Expand on host inventory script authentication Oct 28, 2013
TODO.md Update TODO list Apr 20, 2013

README.md

Puppet Pulse App for Splunk

This app should give insight into the health and functioning of your Puppet installation.

The Views

The views are the following:

  • Overview
  • Changes
  • Statistics
  • Agent Anomalies

The different views should be self-explanatory.

Installation & Setup

Prerequisites

  • A working Puppet master & agents, with logs being indexed by Splunk, either with a Splunk forwarder reading local logs or a central syslog collector.

  • A working Splunk installation.

App Setup

The following are explanations of setup options within Splunk's app setup.

Puppet Host Inventory

The script bin/puppethosts.py connects to the Puppet master's inventory web service to generate a list of known hosts which are managed by the Puppet master. Puppet uses SSL certificate authentication to authenticate Puppet agents and other clients to the Puppet master's inventory service. By default, this script expects to run as root and thereby use the host-agent's private key and certificate and should just work when run that way.

If Splunk is not running as root, other arrangements have to be made to allow the script to authenticate to the Puppet master's inventory web service, such as giving the Splunk user read access to the Puppet agent's private key and certificate, creating a separate certificate in ~splunk/.puppet and signing the resulting cert on the Puppet master, or setting up the command to run as password-less sudo. While these options seem doable, we have not explored them for feasibility.

Also, the Puppet master must be configured to permit the Splunk server access to the /facts endpoint, with an entry in /etc/puppet/auth.conf such as:

path /facts
auth yes
method find, search
allow splunkhost.example.com

More information about configuring Puppet's inventory service can be found at http://docs.puppetlabs.com/guides/inventory_service.html#configuring-access

If you cannot or do not want to use this, you can replace the default, | puppethosts, with something like | inputlookup puppethosts.csv and upload a CSV with columns 'host,fqdn' (where 'host' is the short host name).

Puppet Events

This should be the narrowest search which will include all of your Puppet events -- ideally it will include at least either a source or sourcetype. If your Puppet events are in a non-default index, include this here. All other eventtypes, macros and saved searches (should) use this.

Agent, Master & CIM Log Events

These final searches should differentiate agent, master and cimlog events. I have an extraction for the process name, so am able to match on process=puppet-agent and such, but I am uncertain if the process field was a default extraction or one I configured. If you do not have this, then just matching on 'puppet-agent' or 'puppet-master' should suffice. Note that the agent & master searches should exclude cimlog events.

Other Setup

CIM Log Reporting

Some functionality requires report data from the puppet-cimlog report processor, written to format and log data for ease of consumption by Splunk.

https://github.com/wcooley/puppet-cimlog

Version Reporting

To gather version information, add the following to your manifests (I have it towards the top of my site.pp, outside of any node definitions):

info("node=${::hostname} puppetversion=${::puppetversion}")

Commit Tracking

If your Puppet master config is managed with Git, misc/git-hook-post-update is a post-update hook which logs commits as they enter your repository, which can be useful for narrowing down which commit started causing errors.

To use, copy it to the $MASTER_REPO.git/hooks/ or /etc/puppet/.git/hooks directory of your Puppet master repository and ensure it is executable. (I use the former directory, but I expect that the latter works too, but consider how the commit log time stamps will relate to Puppet events.)

Sample scripts for other VCSs would be a welcome contribution.

Splunk Free vs Enterprise

The free version of Splunk does not include summary indexing, which is used (or will be) to present graphs of long-term data trends. Some manual configuration will likely be necessary, possibly commenting out saved searches using summary-index data and uncommenting searches using directly searched data (also with a much narrower time range).

History

This app started as a copy of the Splunk for Puppet app by Simon Yanick and grew from there. At this point, much has been replaced and eventually all should be.