8 changes: 8 additions & 0 deletions .rubocop.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,14 @@ Bundler/DuplicatedGem:
Enabled: false
Bundler/OrderedGems:
Enabled: false
GetText/DecorateFunctionMessage:
Enabled: false
GetText/DecorateString:
Enabled: false
GetText/DecorateStringFormattingUsingInterpolation:
Enabled: false
GetText/DecorateStringFormattingUsingPercent:
Enabled: false
Layout/AccessModifierIndentation:
Enabled: false
Layout/AlignArray:
Expand Down
25 changes: 21 additions & 4 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
@@ -1,14 +1,31 @@
# Minor Release 5.3.0

## Improvements
- Enable FOSS support with Puppetserver collection
- [PR #23](https://github.com/puppetlabs/puppetlabs-puppet_metrics_collector/pull/23)
- Remove `127.0.0.1` special case naming
- [PR #26](https://github.com/puppetlabs/puppetlabs-puppet_metrics_collector/pull/26)
- Add the ability to generate system metrics
- [PR #28](https://github.com/puppetlabs/puppetlabs-puppet_metrics_collector/pull/28)
- [PR #30](https://github.com/puppetlabs/puppetlabs-puppet_metrics_collector/pull/30)
- Make the output files smaller by excluding metrics and not pretty printing
- [PR #29](https://github.com/puppetlabs/puppetlabs-puppet_metrics_collector/pull/29)

## Changes
- Add a newline to STDOUT of the processing script
- [PR #27](https://github.com/puppetlabs/puppetlabs-puppet_metrics_collector/pull/27)

# Minor Release 5.2.0

## Improvements
- Add ability to send metrics data directly to influxdb, graphite,
or splunk ( in combination with the splunk_hec module )
- https://github.com/puppetlabs/puppetlabs-puppet\_metrics\_collector/pull/19
- Add ability to send metrics data directly to influxdb, graphite,
or splunk ( in combination with the splunk_hec module )
- https://github.com/puppetlabs/puppetlabs-puppet\_metrics\_collector/pull/19

## Changes
- Update metadata dependency information to reflect support for
puppetlabs/stdlib 6.x
- https://github.com/puppetlabs/puppetlabs-puppet\_metrics\_collector/pull/22
- https://github.com/puppetlabs/puppetlabs-puppet\_metrics\_collector/pull/22

# Patch Release 5.1.2

Expand Down
68 changes: 44 additions & 24 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -29,14 +29,27 @@ This module collects metrics provided by the status endpoints of Puppet Enterpri

Install this module with `puppet module install puppetlabs-puppet_metrics_collector` or add it to your Puppetfile.

To activate this module, classify your Primary Master (aka Master of Masters or MoM) with the `puppet_metrics_collector` class using your preferred classification method.
To activate this module, classify your Primary Master (aka Master of Masters or MoM) with the `puppet_metrics_collector` class using your preferred classification method. Below is an example when using the `site.pp`.

```
node 'master.example.com' {
include puppet_metrics_collector
}
```

Optionally, you can also gather some basic system metrics. Unlike the service metrics, this has to be enabled on each host you want metrics from, and the resulting data will be only on that host. Do not include the top level puppet_metrics_collector on anything other than the master as it will collect the same data as the one on the master. This functionality depends on sysstat.

```
node 'master.example.com' {
include puppet_metrics_collector
include puppet_metrics_collector::system
}
node 'compilerA.example.com', 'compilerB.example.com,' {
include puppet_metrics_collector::system
}
```

### Configuration

This module automatically configures the hosts it queries by querying PuppetDB for PE Infrastructure Hosts. If there is an error with automatic configuration of hosts, refer to [Manual Configuration of Hosts](#manual-configuration-of-hosts).
Expand Down Expand Up @@ -95,11 +108,18 @@ Optional String: Allows you to override the command that is run to gather metric

### Grepping Metrics

The metrics come in a JSON hash on one line. In order to convert the metric files into a pretty format, they can be processed with `python -m json.tool` like below.

```bash
cd /opt/puppetlabs/puppet-metrics-collector
for i in <service_name>/master.example.com/*.json; do echo "$(python -m json.tool < $i)" > $i; done
```

You can search for useful information by performing a `grep` in the following format, run from inside the directory containing the metrics.

```
cd /opt/puppetlabs/puppet-metrics-collector
grep <metric_name> <service_name>/127.0.0.1/*.json
grep <metric_name> <service_name>/master.example.com/*.json
```

Since the metrics are compressed every night, you can only search metrics for the current day. To search older metrics, decompress the compressed files into a subdirectory of `/tmp` and run from inside that directory.
Expand All @@ -110,36 +130,36 @@ Since the metrics are compressed every night, you can only search metrics for th
Example:

```
grep average-free-jrubies puppetserver/127.0.0.1/*.json
puppetserver/127.0.0.1/20170404T170501Z.json: "average-free-jrubies": 0.9950009285369501,
puppetserver/127.0.0.1/20170404T171001Z.json: "average-free-jrubies": 0.9999444653324225,
puppetserver/127.0.0.1/20170404T171502Z.json: "average-free-jrubies": 0.9999993830655706,
grep average-free-jrubies puppetserver/master.example.com/*.json
puppetserver/master.example.com/20170404T170501Z.json: "average-free-jrubies": 0.9950009285369501,
puppetserver/master.example.com/20170404T171001Z.json: "average-free-jrubies": 0.9999444653324225,
puppetserver/master.example.com/20170404T171502Z.json: "average-free-jrubies": 0.9999993830655706,
```

#### Grepping PuppetDB Metrics

Example:

```
grep queue_depth puppetdb/127.0.0.1/*.json
puppetdb/127.0.0.1/20170404T170501Z.json: "queue_depth": 0,
puppetdb/127.0.0.1/20170404T171001Z.json: "queue_depth": 0,
puppetdb/127.0.0.1/20170404T171502Z.json: "queue_depth": 0,
grep queue_depth puppetdb/master.example.com/*.json
puppetdb/master.example.com/20170404T170501Z.json: "queue_depth": 0,
puppetdb/master.example.com/20170404T171001Z.json: "queue_depth": 0,
puppetdb/master.example.com/20170404T171502Z.json: "queue_depth": 0,
```

Example for PE 2016.5 and older:

```
grep Cursor puppetdb/127.0.0.1/*.json
puppetdb/127.0.0.1/20170404T171001Z.json: "CursorMemoryUsage": 0,
puppetdb/127.0.0.1/20170404T171001Z.json: "CursorFull": false,
puppetdb/127.0.0.1/20170404T171001Z.json: "CursorPercentUsage": 0,
puppetdb/127.0.0.1/20170404T171502Z.json: "CursorMemoryUsage": 0,
puppetdb/127.0.0.1/20170404T171502Z.json: "CursorFull": false,
puppetdb/127.0.0.1/20170404T171502Z.json: "CursorPercentUsage": 0,
puppetdb/127.0.0.1/20170404T172002Z.json: "CursorMemoryUsage": 0,
puppetdb/127.0.0.1/20170404T172002Z.json: "CursorFull": false,
puppetdb/127.0.0.1/20170404T172002Z.json: "CursorPercentUsage": 0,
grep Cursor puppetdb/master.example.com/*.json
puppetdb/master.example.com/20170404T171001Z.json: "CursorMemoryUsage": 0,
puppetdb/master.example.com/20170404T171001Z.json: "CursorFull": false,
puppetdb/master.example.com/20170404T171001Z.json: "CursorPercentUsage": 0,
puppetdb/master.example.com/20170404T171502Z.json: "CursorMemoryUsage": 0,
puppetdb/master.example.com/20170404T171502Z.json: "CursorFull": false,
puppetdb/master.example.com/20170404T171502Z.json: "CursorPercentUsage": 0,
puppetdb/master.example.com/20170404T172002Z.json: "CursorMemoryUsage": 0,
puppetdb/master.example.com/20170404T172002Z.json: "CursorFull": false,
puppetdb/master.example.com/20170404T172002Z.json: "CursorPercentUsage": 0,
```


Expand Down Expand Up @@ -171,14 +191,14 @@ Example:

```
/opt/puppetlabs/puppet-metrics-collector/puppetserver
├── 127.0.0.1
├── master.example.com
│   ├── 20170404T020001Z.json
│   ├── ...
│ ├── 20170404T170501Z.json
│ └── 20170404T171001Z.json
└── puppetserver-2017.04.04.02.00.01.tar.bz2
/opt/puppetlabs/puppet-metrics-collector/puppetdb
└── 127.0.0.1
└── master.example.com
│ ├── 20170404T020001Z.json
│ ├── ...
│ ├── 20170404T170501Z.json
Expand Down Expand Up @@ -316,13 +336,13 @@ If the future parser is enabled, globally or in the environment, the following c

```
class { 'puppet_metrics_collector':
output_dir => '/opt/puppet/puppet_metrics_collector'
output_dir => '/opt/puppet/puppet-metrics-collector'
}
```

Otherwise, use the following commands.

```
puppet module install puppetlabs-puppet_metrics_collector --modulepath /tmp;
puppet apply -e "class { 'puppet_metrics_collector' : output_dir => '/opt/puppet/puppet_metrics_collector' }" --modulepath /tmp --parser=future
puppet apply -e "class { 'puppet_metrics_collector' : output_dir => '/opt/puppet/puppet-metrics-collector' }" --modulepath /tmp --parser=future
```
Loading