19 changes: 19 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,22 @@
# Major Release 5.0.0

This major release renames the project and obseletes deprecated parameters that
had previously been preserved for backwards compatiblity. The new name of the
project aligns it with the value it provides, and eliminates long incorrect
technology references to its implementation from the name.

Note: If upgrading to puppet\_metrics\_collector 5.x from a
pe\_metric\_curl\_cron\_jobs version older than 4.6.0, it is recommended that
you deploy pe\_metric\_curl\_cron\_jobs 4.6.0 first, let Puppet run, and then
upgrade to puppet\_metrics\_collector 5.x in order to ensure that all cleanup
and migration work is performed smoothly.

## Changes
- Rename the project from "pe\_metric\_curl\_cron\_jobs" to "puppet\_metrics\_collector"
- Remove deprecated parameters
- puppet\_metrics\_collector::puppet\_server\_hosts (long deprecated in favor of puppet\_metrics\_collector::puppetserver\_hosts)
- puppet\_metrics\_collector::puppet\_server\_port (long deprecated in favor of puppet\_metrics\_collector::puppetserver\_port)

# Minor Release 4.6.0

## Improvements:
Expand Down
106 changes: 53 additions & 53 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,77 +31,77 @@ By default, the module tracks the metrics coming from the status endpoint on Pup

## Directory layout

You have a new directory `/opt/puppetlabs/pe_metric_curl_cron_jobs` that has one directory per component (Puppet Server, PuppetDB, or ActiveMQ). Each component has one directory per host that metrics are gathered from. Each host directory contains one JSON file collected every 5 minutes by default. Once per day the metrics for each component are compressed for every host and saved in the root of that component's directory.
You have a new directory `/opt/puppetlabs/puppet-metrics-collector` that has one directory per component (Puppet Server, PuppetDB, or ActiveMQ). Each component has one directory per host that metrics are gathered from. Each host directory contains one JSON file collected every 5 minutes by default. Once per day the metrics for each component are compressed for every host and saved in the root of that component's directory.

Here's an example:

~~~
/opt/puppetlabs/pe_metric_curl_cron_jobs/puppetserver
```
/opt/puppetlabs/puppet-metrics-collector/puppetserver
├── 127.0.0.1
│   ├── 20170404T020001Z.json
│   ├── ...
│ ├── 20170404T170501Z.json
│ └── 20170404T171001Z.json
└── puppetserver-2017.04.04.02.00.01.tar.bz2
/opt/puppetlabs/pe_metric_curl_cron_jobs/puppetdb
/opt/puppetlabs/puppet-metrics-collector/puppetdb
└── 127.0.0.1
│ ├── 20170404T020001Z.json
│ ├── ...
│ ├── 20170404T170501Z.json
│ ├── 20170404T171001Z.json
└── puppetdb-2017.04.04.02.00.01.tar.bz2
~~~
```

## Cron jobs

Each component has two cron jobs created for it.

- A cron job to gather the metrics
- Runs every 5 minutes
- A cron job to delete metrics past the rentention_days and compress metrics
- A cron job to delete metrics past the rentention\_days and compress metrics
- Runs at randomly selected time between midnight and 3AM

Example:

~~~
```
crontab -l
...
# Puppet Name: puppetserver_metrics_collection
*/5 * * * * /opt/puppetlabs/pe_metric_curl_cron_jobs/scripts/puppetserver_metrics
*/5 * * * * /opt/puppetlabs/puppet-metrics-collector/scripts/puppetserver_metrics
# Puppet Name: puppetserver_metrics_tidy
0 2 * * * /opt/puppetlabs/pe_metric_curl_cron_jobs/scripts/puppetserver_metrics_tidy
~~~
0 2 * * * /opt/puppetlabs/puppet-metrics-collector/scripts/puppetserver_metrics_tidy
```

## Grepping for Metrics

You can get useful information with a grep like the one below run from inside of the directory containing the metrics files. Since the metrics are compressed every night you can only grep metrics for the current day. If you'd like to grep over a longer period of time you should decompress the compressed tarballs into `/tmp` and investigate further.

~~~
cd /opt/puppetlabs/pe_metric_curl_cron_jobs
```
cd /opt/puppetlabs/puppet-metrics-collector
grep <metric_name> <component_name>/127.0.0.1/*.json
~~~
```

### Puppetserver

Example output:

~~~
```
grep average-free-jrubies puppetserver/127.0.0.1/*.json
puppetserver/127.0.0.1/20170404T170501Z.json: "average-free-jrubies": 0.9950009285369501,
puppetserver/127.0.0.1/20170404T171001Z.json: "average-free-jrubies": 0.9999444653324225,
puppetserver/127.0.0.1/20170404T171502Z.json: "average-free-jrubies": 0.9999993830655706,
~~~
```

### PuppetDB

Example output:

~~~
```
grep queue_depth puppetdb/127.0.0.1/*.json
puppetdb/127.0.0.1/20170404T170501Z.json: "queue_depth": 0,
puppetdb/127.0.0.1/20170404T171001Z.json: "queue_depth": 0,
puppetdb/127.0.0.1/20170404T171502Z.json: "queue_depth": 0,
~~~
```

PE 2016.5 and below:

Expand Down Expand Up @@ -133,9 +133,9 @@ The script creates a tarball containing your metrics in the current working dire

# How to use

Install the module with `puppet module install npwalker-pe_metric_curl_cron_jobs` or add it to your Puppetfile.
Install the module with `puppet module install puppetlabs-puppet_metrics_collector` or add it to your Puppetfile.

To start data collection you will need to classify your puppet master with the `pe_metric_curl_cron_jobs` class using your preferred classification method.
To start data collection you will need to classify your puppet master with the `puppet_metrics_collector` class using your preferred classification method.

The following examples show how to configure the parameters to work in different setups but we assume you will always classify on the node that is the CA master. The preferred method is to `include` the module and then provide hiera
data for the parameters.
Expand All @@ -148,103 +148,103 @@ None needed for a monolithic install

### Class Definition

~~~
include pe_metric_curl_cron_jobs
~~~
```
include puppet_metrics_collector
```

## Split Install ( Running on the Master )

### Hiera data Example

~~~
pe_metric_curl_cron_jobs::puppetdb_hosts:
```
puppet_metrics_collector::puppetdb_hosts:
- 'split-puppetdb.domain.com'
~~~
```

### Class Definition Example

~~~
class { 'pe_metric_curl_cron_jobs':
```
class { 'puppet_metrics_collector':
puppetdb_hosts => ['split-puppetdb.domain.com']
}
~~~
```

## Monolithic With Compile Masters ( Running on the MoM )

### Hiera data Example

~~~
pe_metric_curl_cron_jobs::puppetserver_hosts:
```
puppet_metrics_collector::puppetserver_hosts:
- 'master-1.domain.com'
- 'compile-master-1.domain.com'
- 'compile-master-2.domain.com'
~~~
```

### Class Definition Example

~~~
class { 'pe_metric_curl_cron_jobs':
```
class { 'puppet_metrics_collector':
puppetserver_hosts => [
'master-1.domain.com',
'compile-master-1.domain.com',
'compile-master-2.domain.com'
]
}
~~~
```

## Split With Compile Masters ( Running on the MoM )

### Hiera data Example

~~~
pe_metric_curl_cron_jobs::puppetdb_hosts:
```
puppet_metrics_collector::puppetdb_hosts:
- 'split-puppetdb.domain.com'
pe_metric_curl_cron_jobs::puppetserver_hosts:
puppet_metrics_collector::puppetserver_hosts:
- 'master-1.domain.com'
- 'compile-master-1.domain.com'
- 'compile-master-2.domain.com'
~~~
```

### Class Definition Example

~~~
class { 'pe_metric_curl_cron_jobs':
```
class { 'puppet_metrics_collector':
puppetdb_hosts => ['split-puppetdb.domain.com'],
puppetserver_hosts => [
'master-1.domain.com',
'compile-master-1.domain.com',
'compile-master-2.domain.com'
]
}
~~~
```

## Running on PE 3.8

You can still use this module on PE 3.8 although you have to run it with the future parser and you want to use `/opt/puppet` instead of `/opt/puppetlabs`. If the [future parser](https://docs.puppet.com/puppet/3.8/experiments_future.html) is enabled in the environment or globally, the following can be put in the site.pp.

~~~
class { 'pe_metric_curl_cron_jobs':
output_dir => '/opt/puppet/pe_metric_curl_cron_jobs'
```
class { 'puppet_metrics_collector':
output_dir => '/opt/puppet/puppet_metrics_collector'
}
~~~
```

The module can be run in a one off run if the future parser is not enabled in the environment.

~~~
puppet module install npwalker-pe_metric_curl_cron_jobs --modulepath /tmp;
puppet apply -e "class { 'pe_metric_curl_cron_jobs' : output_dir => '/opt/puppet/pe_metric_curl_cron_jobs' }" --modulepath /tmp --parser=future
~~~
```
puppet module install puppetlabs-puppet_metrics_collector --modulepath /tmp;
puppet apply -e "class { 'puppet_metrics_collector' : output_dir => '/opt/puppet/puppet_metrics_collector' }" --modulepath /tmp --parser=future
```

If you do not want to manage this long term and want to get it up and running quickly you can run it via puppet apply. Make sure the puppetlabs-stdlib module is installed. Refer to the other examples if you want to change other parameters.

## Temporary Install

The module installation is the best way to utilize this module, but it can be run on a one off basis with the following command.

~~~
puppet module install npwalker-pe_metric_curl_cron_jobs --modulepath /tmp;
puppet apply -e "class { 'pe_metric_curl_cron_jobs': }" --modulepath /tmp;
~~~
```
puppet module install puppetlabs-puppet_metrics_collector --modulepath /tmp;
puppet apply -e "class { 'puppet_metrics_collector': }" --modulepath /tmp;
```

## Alternate Option for Multi-node Metrics Collection

Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
require 'yaml'

Puppet::Functions.create_function(:'pe_metric_curl_cron_jobs::to_yaml') do
Puppet::Functions.create_function(:'puppet_metrics_collector::to_yaml') do
dispatch :to_yaml do
param 'Hash', :hash_or_array
end
Expand Down
22 changes: 11 additions & 11 deletions manifests/activemq.pp
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
class pe_metric_curl_cron_jobs::activemq (
Integer $collection_frequency = $::pe_metric_curl_cron_jobs::collection_frequency,
Integer $retention_days = $::pe_metric_curl_cron_jobs::retention_days,
String $metrics_ensure = $::pe_metric_curl_cron_jobs::activemq_metrics_ensure,
Array[String] $hosts = $::pe_metric_curl_cron_jobs::activemq_hosts,
Integer $port = $::pe_metric_curl_cron_jobs::activemq_port,
class puppet_metrics_collector::activemq (
Integer $collection_frequency = $puppet_metrics_collector::collection_frequency,
Integer $retention_days = $puppet_metrics_collector::retention_days,
String $metrics_ensure = $puppet_metrics_collector::activemq_metrics_ensure,
Array[String] $hosts = $puppet_metrics_collector::activemq_hosts,
Integer $port = $puppet_metrics_collector::activemq_port,
) {
$scripts_dir = $::pe_metric_curl_cron_jobs::scripts_dir
$scripts_dir = $::puppet_metrics_collector::scripts_dir

Pe_metric_curl_cron_jobs::Pe_metric {
output_dir => $::pe_metric_curl_cron_jobs::output_dir,
Puppet_metrics_collector::Pe_metric {
output_dir => $::puppet_metrics_collector::output_dir,
scripts_dir => $scripts_dir,
cron_minute => "*/${collection_frequency}",
retention_days => $retention_days,
Expand Down Expand Up @@ -55,10 +55,10 @@
file { "${scripts_dir}/amq_metrics" :
ensure => present,
mode => '0744',
source => 'puppet:///modules/pe_metric_curl_cron_jobs/amq_metrics',
source => 'puppet:///modules/puppet_metrics_collector/amq_metrics',
}

pe_metric_curl_cron_jobs::pe_metric { 'activemq' :
puppet_metrics_collector::pe_metric { 'activemq' :
metric_ensure => $metrics_ensure,
hosts => $hosts,
metrics_port => $port,
Expand Down
43 changes: 20 additions & 23 deletions manifests/init.pp
Original file line number Diff line number Diff line change
@@ -1,13 +1,9 @@
class pe_metric_curl_cron_jobs (
# DEPRECATED API ==============================
Optional[String] $puppet_server_metrics_ensure = undef,
Optional[Array[String]] $puppet_server_hosts = undef,
# CURRENT API =================================
String $output_dir = '/opt/puppetlabs/pe_metric_curl_cron_jobs',
class puppet_metrics_collector (
String $output_dir = '/opt/puppetlabs/puppet-metrics-collector',
Integer $collection_frequency = 5,
Integer $retention_days = 90,
String $puppetserver_metrics_ensure = pick($pe_metric_curl_cron_jobs::puppet_server_metrics_ensure, 'present'),
Array[String] $puppetserver_hosts = pick($pe_metric_curl_cron_jobs::puppet_server_hosts, [ '127.0.0.1' ]),
String $puppetserver_metrics_ensure = present,
Array[String] $puppetserver_hosts = [ '127.0.0.1' ],
Integer $puppetserver_port = 8140,
String $puppetdb_metrics_ensure = 'present',
Array[String] $puppetdb_hosts = [ '127.0.0.1' ],
Expand All @@ -30,15 +26,15 @@
file { "${scripts_dir}/tk_metrics" :
ensure => present,
mode => '0755',
source => 'puppet:///modules/pe_metric_curl_cron_jobs/tk_metrics'
source => 'puppet:///modules/puppet_metrics_collector/tk_metrics'
}

file { "${bin_dir}/puppet-metrics-collector":
ensure => file,
owner => 'root',
group => 'root',
mode => '0755',
content => epp('pe_metric_curl_cron_jobs/puppet-metrics-collector.epp', {
content => epp('puppet_metrics_collector/puppet-metrics-collector.epp', {
'output_dir' => $output_dir,
}),
}
Expand All @@ -53,20 +49,21 @@
target => "${bin_dir}/puppet-metrics-collector",
}

include pe_metric_curl_cron_jobs::puppetserver
include puppet_metrics_collector::puppetserver
include puppet_metrics_collector::puppetdb
include puppet_metrics_collector::orchestrator
include puppet_metrics_collector::activemq

include pe_metric_curl_cron_jobs::puppetdb
# LEGACY CLEANUP
# This exec resource exists to clean up old metrics directories created by
# the module before it was renamed.
$legacy_dir = '/opt/puppetlabs/pe_metric_curl_cron_jobs'
$safe_output_dir = shellquote($output_dir)

include pe_metric_curl_cron_jobs::orchestrator

include pe_metric_curl_cron_jobs::activemq

# Emit deprecation warnings if necessary
if ($puppet_server_metrics_ensure != undef) {
warning('Using deprecated parameter pe_metric_curl_cron_jobs::puppet_server_metrics_ensure! please use pe_metric_curl_cron_jobs::puppetserver_metrics_ensure instead.')
exec { "migrate ${legacy_dir} directory":
path => '/bin:/usr/bin',
command => "mv ${legacy_dir} ${safe_output_dir}",
onlyif => "[ ! -e ${safe_output_dir} -a -e ${legacy_dir} ]",
before => File[$output_dir],
}
if ($puppet_server_hosts != undef) {
warning('Using deprecated parameter pe_metric_curl_cron_jobs::puppet_server_hosts! please use pe_metric_curl_cron_jobs::puppetserver_hosts instead.')
}

}
Loading