24 changes: 24 additions & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# Major Release 3.0.0

## Changes
- Every parameter, file name, etc... that contained puppet_server is rewritten
to puppetserver
- The existing parameters remain but are deprecated and should not be used
- Metric storage format is a single JSON blob instead of the exact output from
whichever endpoint was queried

## Improvements
- Metrics gathering scripts are rewritten in ruby
- Metrics are now stored in one file per component
- PuppetDB metrics were previously stored with one file per metric
- Metrics are now stored in one directory per server
- PuppetDB metrics now gathers the status endpoint
- This is the preferred way to get the queue_depth metric
- Opt-in collection of ActiveMQ metrics is available
- Metrics are compressed daily for a 90% reduction in disk space
- Metrics are retained for 90 days by default instead of 3 days
- Retained metrics still take less space due to compression savings

## Bug Fixes:
- The metrics tidy cron job previously ran every minute between 2-3 AM.
It now runs just once at 2AM.
322 changes: 205 additions & 117 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,154 +1,242 @@
Table of Contents
=================

* [How to use](#how-to-use)
* [Monolithic Install](#monolithic-install)
* [Split Install ( Running on the Master )](#split-install--running-on-the-master-)
* [Monolithic With Compile Masters ( Running on the MoM )](#monolithic-with-compile-masters--running-on-the-mom-)
* [Split With Compile Masters ( Running on the MoM )](#split-with-compile-masters--running-on-the-mom-)
* [Running on PE 3\.8](#running-on-pe-38)
* [Other Option](#other-option)
* [What do you get](#what-do-you-get)
* [Grepping for Metrics](#grepping-for-metrics)
* [Puppetserver](#puppetserver)
* [PuppetDB](#puppetdb)
* [What do you get](#what-do-you-get)
* [Directory layout](#directory-layout)
* [Cron jobs](#cron-jobs)
* [Grepping for Metrics](#grepping-for-metrics)
* [Puppetserver](#puppetserver)
* [PuppetDB](#puppetdb)
* [How to use](#how-to-use)
* [Monolithic Install](#monolithic-install)
* [Hiera data Example](#hiera-data-example)
* [Class Definition](#class-definition)
* [Split Install ( Running on the Master )](#split-install--running-on-the-master-)
* [Hiera data Example](#hiera-data-example-1)
* [Class Definition Example](#class-definition-example)
* [Monolithic With Compile Masters ( Running on the MoM )](#monolithic-with-compile-masters--running-on-the-mom-)
* [Hiera data Example](#hiera-data-example-2)
* [Class Definition Example](#class-definition-example-1)
* [Split With Compile Masters ( Running on the MoM )](#split-with-compile-masters--running-on-the-mom-)
* [Hiera data Example](#hiera-data-example-3)
* [Class Definition Example](#class-definition-example-2)
* [Running on PE 3\.8](#running-on-pe-38)
* [Temporary Install](#temporary-install)
* [Alternate Option for Multi\-node Metrics Collection](#alternate-option-for-multi-node-metrics-collection)

# What do you get

By default, the module tracks the metrics coming from the status endpoint on Puppet Server and PuppetDB as well as a curated set of metrics from PuppetDB.

## Directory layout

You have a new directory `/opt/puppetlabs/pe_metric_curl_cron_jobs` that has one directory per component (Puppet Server, PuppetDB, or ActiveMQ). Each component has one directory per host that metrics are gathered from. Each host directory contains one JSON file collected every 5 minutes by default. Once per day the metrics for each component are compressed for every host and saved in the root of that component's directory.

Here's an example:

~~~
/opt/puppetlabs/pe_metric_curl_cron_jobs/puppetserver
├── 127.0.0.1
│   ├── 20170404T020001Z.json
│   ├── ...
│ ├── 20170404T170501Z.json
│ └── 20170404T171001Z.json
└── puppetserver-2017.04.04.02.00.01.tar.bz2
/opt/puppetlabs/pe_metric_curl_cron_jobs/puppetdb
└── 127.0.0.1
│ ├── 20170404T020001Z.json
│ ├── ...
│ ├── 20170404T170501Z.json
│ ├── 20170404T171001Z.json
└── puppetdb-2017.04.04.02.00.01.tar.bz2
~~~

## Cron jobs

Each component has two cron jobs created for it.

- A cron job to gather the metrics
- Runs every 5 minutes
- A cron job to delete metrics past the rentention_days and compress metrics
- Runs daily at 2AM

Example:

~~~
crontab -l
...
# Puppet Name: puppetserver_metrics_collection
*/5 * * * * /opt/puppetlabs/pe_metric_curl_cron_jobs/scripts/puppetserver_metrics
# Puppet Name: puppetserver_metrics_tidy
0 2 * * * /opt/puppetlabs/pe_metric_curl_cron_jobs/scripts/puppetserver_metrics_tidy
~~~

## Grepping for Metrics

You can get useful information with a grep like the one below run from inside of the directory containing the metrics files. Since the metrics are compressed every night you can only grep metrics for the current day. If you'd like to grep over a longer period of time you should decompress the compressed tarballs into `/tmp` and investigate further.

~~~
cd /opt/puppetlabs/pe_metric_curl_cron_jobs
grep <metric_name> <component_name>/127.0.0.1/*.json
~~~

### Puppetserver

Example output:

~~~
grep average-free-jrubies puppetserver/127.0.0.1/*.json
puppetserver/127.0.0.1/20170404T170501Z.json: "average-free-jrubies": 0.9950009285369501,
puppetserver/127.0.0.1/20170404T171001Z.json: "average-free-jrubies": 0.9999444653324225,
puppetserver/127.0.0.1/20170404T171502Z.json: "average-free-jrubies": 0.9999993830655706,
~~~

### PuppetDB

Example output:

~~~
grep queue_depth puppetdb/127.0.0.1/*.json
puppetdb/127.0.0.1/20170404T170501Z.json: "queue_depth": 0,
puppetdb/127.0.0.1/20170404T171001Z.json: "queue_depth": 0,
puppetdb/127.0.0.1/20170404T171502Z.json: "queue_depth": 0,
~~~

PE 2016.5 and below:

~~~
grep Cursor puppetdb/127.0.0.1/*.json
puppetdb/127.0.0.1/20170404T171001Z.json: "CursorMemoryUsage": 0,
puppetdb/127.0.0.1/20170404T171001Z.json: "CursorFull": false,
puppetdb/127.0.0.1/20170404T171001Z.json: "CursorPercentUsage": 0,
puppetdb/127.0.0.1/20170404T171502Z.json: "CursorMemoryUsage": 0,
puppetdb/127.0.0.1/20170404T171502Z.json: "CursorFull": false,
puppetdb/127.0.0.1/20170404T171502Z.json: "CursorPercentUsage": 0,
puppetdb/127.0.0.1/20170404T172002Z.json: "CursorMemoryUsage": 0,
puppetdb/127.0.0.1/20170404T172002Z.json: "CursorFull": false,
puppetdb/127.0.0.1/20170404T172002Z.json: "CursorPercentUsage": 0,
~~~

# How to use

```
include pe_metric_curl_cron_jobs
```
Install the module with `puppet module install npwalker-pe_metric_curl_cron_jobs` or add it to your Puppetfile.

To start data collection you will need to classify your puppet master with the `pe_metric_curl_cron_jobs` class using your preferred classification method.

If you do not want to manage this long term and want to get it up and running quickly you can run it via puppet apply.
The following examples show how to configure the parameters to work in different setups but we assume you will always classify on the node that is the CA master. The preferred method is to `include` the module and then provide hiera
data for the parameters.

## Monolithic Install

```
cd /tmp;
git clone https://github.com/npwalker/pe_metric_curl_cron_jobs;
puppet apply -e "class { 'pe_metric_curl_cron_jobs': }" --modulepath .
```
### Hiera data Example

## Split Install ( Running on the Master )
None needed for a monolithic install

```
cd /tmp;
git clone https://github.com/npwalker/pe_metric_curl_cron_jobs;
puppet apply -e "class { 'pe_metric_curl_cron_jobs' : puppetdb_hosts => ['split-puppetdb.domain.com'] }" --modulepath .
```
### Class Definition

## Monolithic With Compile Masters ( Running on the MoM )
~~~
include pe_metric_curl_cron_jobs
~~~

```
cd /tmp;
git clone https://github.com/npwalker/pe_metric_curl_cron_jobs;
puppet apply -e "class { 'pe_metric_curl_cron_jobs' : puppet_server_hosts => ['compile-master-1.domain.com', 'compile-master-2.domain.com'] }" --modulepath .
```
## Split Install ( Running on the Master )

## Split With Compile Masters ( Running on the MoM )
### Hiera data Example

```
cd /tmp;
git clone https://github.com/npwalker/pe_metric_curl_cron_jobs;
puppet apply -e "class { 'pe_metric_curl_cron_jobs' : puppetdb_hosts => ['split-puppetdb.domain.com'], puppet_server_hosts => ['compile-master-1.domain.com', 'compile-master-2.domain.com'] }" --modulepath .
```
~~~
pe_metric_curl_cron_jobs::puppetdb_hosts:
- 'split-puppetdb.domain.com'
~~~

## Running on PE 3.8
### Class Definition Example

You can still use this module on PE 3.8 although you have to run it with the future parser and you want to use `/opt/puppet` instead of `/opt/puppetlabs`.
~~~
class { 'pe_metric_curl_cron_jobs':
puppetdb_hosts => ['split-puppetdb.domain.com']
}
~~~

```
cd /tmp;
git clone https://github.com/npwalker/pe_metric_curl_cron_jobs;
puppet apply -e "class { 'pe_metric_curl_cron_jobs' : output_dir => '/opt/puppet/pe_metric_curl_cron_jobs' }" --modulepath . --parser=future
```
## Monolithic With Compile Masters ( Running on the MoM )

Refer to the other examples if you want to change other parameters.
### Hiera data Example

## Other Option
~~~
pe_metric_curl_cron_jobs::puppetserver_hosts:
- 'master-1.domain.com'
- 'compile-master-1.domain.com'
- 'compile-master-2.domain.com'
~~~

This option puts metrics on each individual node so I don't think it's as good as having centrally gathered metrics but you can also install the module on each individual node. If you install on a compile master you can set `puppetdb_metrics_ensure` to `absent` and if you install on a puppetdb node then you can set `$puppet_server_metrics_ensure` to `absent`.
### Class Definition Example

# What do you get
~~~
class { 'pe_metric_curl_cron_jobs':
puppetserver_hosts => [
'master-1.domain.com',
'compile-master-1.domain.com',
'compile-master-2.domain.com'
]
}
~~~

By default the module tracks the metrics coming from the status endpoint on Puppetserver and the internal ActiveMQ metrics on PuppetDB.
## Split With Compile Masters ( Running on the MoM )

A new directory `/opt/puppetlabs/pe_metric_curl_cron_jobs` that looks like:
### Hiera data Example

~~~
pe_metric_curl_cron_jobs::puppetdb_hosts:
- 'split-puppetdb.domain.com'
pe_metric_curl_cron_jobs::puppetserver_hosts:
- 'master-1.domain.com'
- 'compile-master-1.domain.com'
- 'compile-master-2.domain.com'
~~~

### Class Definition Example

~~~
class { 'pe_metric_curl_cron_jobs':
puppetdb_hosts => ['split-puppetdb.domain.com'],
puppetserver_hosts => [
'master-1.domain.com',
'compile-master-1.domain.com',
'compile-master-2.domain.com'
]
}
~~~

```
/opt/puppetlabs/pe_metric_curl_cron_jobs/
├── puppetdb
│   ├── 127.0.0.1-08_16_16_23:40.json
│   └── 127.0.0.1-08_16_16_23:45.json
│   └── 127.0.0.1-08_16_16_23:50.json
├── puppet_server
│   ├── 127.0.0.1-08_16_16_23:40.json
│   ├── 127.0.0.1-08_16_16_23:45.json
│   ├── 127.0.0.1-08_16_16_23:50.json
└── scripts
├── puppetdb_metrics.sh
└── puppet_server_metrics.sh
```
## Running on PE 3.8

New cronjobs:
You can still use this module on PE 3.8 although you have to run it with the future parser and you want to use `/opt/puppet` instead of `/opt/puppetlabs`. If the [future parser](https://docs.puppet.com/puppet/3.8/experiments_future.html) is enabled in the environment or globally, the following can be put in the site.pp.

```
crontab -l
...
# Puppet Name: puppet_server_metrics_collection
*/5 * * * * /opt/puppetlabs/pe_metric_curl_cron_jobs/scripts/puppet_server_metrics.sh
# Puppet Name: puppet_server_metrics_tidy
* 2 * * * puppet apply -e " tidy { '/opt/puppetlabs/pe_metric_curl_cron_jobs/puppet_server' : age => '3d', recurse => 1 } "
# Puppet Name: puppetdb_metrics_collection
*/5 * * * * /opt/puppetlabs/pe_metric_curl_cron_jobs/scripts/puppetdb_metrics.sh
# Puppet Name: puppetdb_metrics_tidy
* 2 * * * puppet apply -e " tidy { '/opt/puppetlabs/pe_metric_curl_cron_jobs/puppetdb' : age => '3d', recurse => 1 } "
```
~~~
class { 'pe_metric_curl_cron_jobs':
output_dir => '/opt/puppet/pe_metric_curl_cron_jobs'
}
~~~

## Grepping for Metrics
The module can be run in a one off run if the future parser is not enabled in the environment.

You can get useful information with a grep like the one below run from inside of the directory containing the metrics files.
~~~
puppet module install npwalker-pe_metric_curl_cron_jobs --modulepath /tmp;
puppet apply -e "class { 'pe_metric_curl_cron_jobs' : output_dir => '/opt/puppet/pe_metric_curl_cron_jobs' }" --modulepath /tmp --parser=future
~~~

```
grep <metric_name> 127.0.0.1-*
```
If you do not want to manage this long term and want to get it up and running quickly you can run it via puppet apply. Make sure the puppetlabs-stdlib module is installed. Refer to the other examples if you want to change other parameters.

### Puppetserver
## Temporary Install

Example output:
The module installation is the best way to utilize this module, but it can be run on a one off basis with the following command.

```
grep average-free-jrubies 127.0.0.1-*
127.0.0.1-08_15_16_10:55.json: "average-free-jrubies": 3.6687597774999556,
127.0.0.1-08_15_16_11:00.json: "average-free-jrubies": 4.4209186472147248,
127.0.0.1-08_15_16_11:05.json: "average-free-jrubies": 3.610399319630555,
127.0.0.1-08_15_16_11:10.json: "average-free-jrubies": 4.9845629308522383,
```
~~~
puppet module install npwalker-pe_metric_curl_cron_jobs --modulepath /tmp;
puppet apply -e "class { 'pe_metric_curl_cron_jobs': }" --modulepath /tmp;
~~~

### PuppetDB
## Alternate Option for Multi-node Metrics Collection

Example output:
This option puts metrics on each individual node instead of gathering metrics centrally on the CA master. In order to do so, you would classify each of your PE infrastructure nodes with this module. This option is discouraged but if
for some reason you can't reach out across a network segment or some other reason you may still wish to have metrics.

When you classify a compile master you would set `$puppetdb_metrics_ensure` to `absent`.

```
grep QueueSize 127.0.0.1-*
127.0.0.1-08_16_16_23:40.json: "QueueSize" : 0,
127.0.0.1-08_16_16_23:45.json: "QueueSize" : 0,
```

```
grep CursorMemoryUsage 127.0.0.1-*
127.0.0.1-08_16_16_23:40.json: "CursorMemoryUsage" : 0,
127.0.0.1-08_16_16_23:45.json: "CursorMemoryUsage" : 0,
```

```
grep CursorFull 127.0.0.1-*
127.0.0.1-08_16_16_23:40.json: "CursorFull" : false,
127.0.0.1-08_16_16_23:45.json: "CursorFull" : false,
```

```
grep CursorPercentUsage 127.0.0.1-*
127.0.0.1-08_16_16_23:40.json: "CursorPercentUsage" : 0,
127.0.0.1-08_16_16_23:45.json: "CursorPercentUsage" : 0,
```
When you classify a PuppetDB node you would set `$puppetserver_metrics_ensure` to `absent`.
Loading