211 changes: 64 additions & 147 deletions ARCHIVES.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,179 +2,96 @@

## Requirements

Telegraf can be run manually to import archives from the `puppet_metrics_collector` module to InfluxDB. To do so, you'll need the following:
This module offers Bolt plans to provision a non-SSL dashboard node and to import archive metrics. You will need a [Bolt installation](https://puppet.com/docs/bolt/latest/bolt_installing.html) on a client node and access to the remote node you wish to target.

* An InfluxDB [organization](https://github.com/puppetlabs/puppet_operational_dashboards/blob/main/REFERENCE.md#initial_org) and [bucket](https://github.com/puppetlabs/puppet_operational_dashboards/blob/main/REFERENCE.md#initial_bucket). This module's defaults are `puppetlabs` and `puppet_data` respectively, but these can be configured via the linked parameters.

The sample [bucket_and_datasource class](https://github.com/puppetlabs/puppet_operational_dashboards/blob/main/examples/bucket_and_datasource.txt) shows how to create a bucket and Grafana datasource ad hoc using `puppet apply`. See [#examples](#examples) for sample usage.

* A token with permissions to write to the bucket. This can be the admin token created by the [puppetlabs-influxdb module](https://github.com/puppetlabs/influxdb/blob/main/REFERENCE.md#token_file), or the [Telegraf token](https://github.com/puppetlabs/puppet_operational_dashboards/blob/main/REFERENCE.md#telegraf_token_name) created by this module. For more information, see [what-puppet_operational_dashboards-affects](https://github.com/puppetlabs/puppet_operational_dashboards/blob/main/README.md#what-puppet_operational_dashboards-affects).

The token will need to be exported to the environment. Prefacing this command with a space will keep it out of shell history:
## Setup

* Install Bolt and clone this module
```
export INFLUX_TOKEN=<token>
git clone https://github.com/puppetlabs/puppet_operational_dashboards.git
```

* A set of Telegraf configuration files and scripts to process and emit the metrics.

A sample [telegraf.conf](https://github.com/puppetlabs/puppet_operational_dashboards/tree/main/examples/telegraf.conf) to configure the application and [telegraf.conf.d](https://github.com/puppetlabs/puppet_operational_dashboards/tree/main/examples/telegraf.conf.d) directory to process the metrics are provided.

`telegraf.conf` will need to point to the bucket and url of your InfluxDB server. Change `<my_bucket>` and `<influxdb_fqdn>` as needed. See [examples](#examples) for sample usage.

* Valid SSL certs to talk to your InfluxDB server, if using SSL.

`puppetlabs-influxdb` defaults to configuring InfluxDB with SSL and uses certs issued by the Puppet CA. In addition to configuring the InfluxDB application, it will copy the necessary certs to `/etc/telegraf` for Telegraf to communicate using SSL. Therefore, if you are importing archive metrics on a node configured with the [puppet_operational_dashboards::telegraf::agent](https://github.com/puppetlabs/puppet_operational_dashboards/blob/main/REFERENCE.md#puppet_operational_dashboardstelegrafagent) class, this will already be in place. If not, such as if you are importing archives from an unmanaged workstation, you will need the following:

An [SSL cert](https://github.com/puppetlabs/influxdb/blob/main/REFERENCE.md#ssl_cert_file) saved to `/etc/telegraf/cert.pem`.

A [CA cert](https://github.com/puppetlabs/influxdb/blob/main/REFERENCE.md#ssl_ca_file) saved to `/etc/telegraf/ca.pem`.


## Examples

These examples assume an organization named `puppetlabs`, a bucket named `puppet_data`, and an InfluxDB server with an fqdn of `influx-host`.

### Telegraf configuration

* Save [telegraf.conf](https://github.com/puppetlabs/puppet_operational_dashboards/tree/main/examples/telegraf.conf) to your home directory `~/telegraf.conf` and configure it:

* Install the required modules
```
[agent]
hostname = ""
omit_hostname = false
interval = "10m"
round_interval = true
metric_batch_size = 10000
metric_buffer_limit = 1000000
collection_jitter = "0s"
flush_interval = "5m"
flush_jitter = "0s"
precision = ""
logfile = ""
debug = false
quiet = false
[[outputs.influxdb_v2]]
bucket = "puppet_data"
organization = "puppetlabs"
tls_ca = "/etc/telegraf/ca.pem"
tls_cert = "/etc/telegraf/cert.pem"
token = "$INFLUX_TOKEN"
urls = ["https://influx-host:8086"]
cd puppet_operational_dashboards
bolt module install
```

* Save the files in [telegraf.conf.d](https://github.com/puppetlabs/puppet_operational_dashboards/tree/main/examples/telegraf.conf.d) to your home directory `~/telegraf.conf.d`. These don't require configuration unless the structure of your metrics files differs from the steps below.

### Metrics extraction

If you are using a metrics archive from the [metrics collector](https://github.com/puppetlabs/puppetlabs-puppet_metrics_collector/#sharing-metrics-data), create a directory named `metrics` and extract the archive to it.

* Install the toml-rb gem
```
mkdir metrics
tar xf <metrics_archives> -C metrics --strip-components 1
/opt/puppetlabs/bolt/bin/gem install --user-install toml-rb
```

If using a PE support script, extract it and change to its top level directory.
## Usage

### puppet_operational_dashboards::provision_dashboard
```
tar xf <pe_support_script>.tar.gz
cd <pe_support_script>
bolt plan run puppet_operational_dashboards::provision_dashboard targets=<value>
Parameters
targets TargetSpec
The targets to run on.
```

Next, extract all individual archive files in the `metrics` directory:
### puppet_operational_dashboards::load_metrics

The only mandatory parameter is `metrics_dir`, the path to the metrics contained in a PE support script.
```
find metrics/ -type f -name "*gz" -execdir tar xf "{}" \;
```
bolt plan run puppet_operational_dashboards::load_metrics
[cleanup_metrics=<value>] [conf_dir=<value>] [dest_dir=<value>]
[grafana_datasource=<value>] [influxdb_bucket=<value>] [influxdb_org=<value>]
[influxdb_port=<value>] metrics_dir=<value> targets=<value>
[telegraf_process=<value>] [telegraf_token=<value>] [token=<value>]
[token_file=<value>]
Lastly, delete any Puppet server metrics with errors. Currently, these will cause the `telegraf` process to exit upon encountering an error. Delete these with:
```
find metrics/puppetserver -type f -name "*json" -size -1000c -delete
```
Parameters
cleanup_metrics String
Whether to delete metrics after processing
Default: 'true'
### Create an InfluxDB bucket and Grafana datasource
conf_dir String
Directory to upload Telegraf configuration files to
Default: '/tmp/telegraf'
You may skip this step if using the default [organization](https://github.com/puppetlabs/puppet_operational_dashboards/blob/main/REFERENCE.md#initial_org) and [bucket](https://github.com/puppetlabs/puppet_operational_dashboards/blob/main/REFERENCE.md#initial_bucket) from this module, or otherwise have already created them.
dest_dir String
Directory to upload $metrics_dir to
Default: '/tmp'
An organization and bucket may be continuously enforced as part of a catalog compilation using resources from the `puppetlabs-influxdb` module. See the [influxdb_org](https://github.com/puppetlabs/influxdb/blob/main/REFERENCE.md#influxdb_org) and [influxdb_bucket](https://github.com/puppetlabs/influxdb/blob/main/REFERENCE.md#influxdb_bucket) resources for more.
grafana_datasource String
Name of the Grafana datasource. Must match the name of the InfluxDB bucket
Default: $influxdb_bucket
Alternatively, the [bucket_and_datasource class](https://github.com/puppetlabs/puppet_operational_dashboards/blob/main/examples/bucket_and_datasource.txt) shows how to create a bucket and Grafana datasource in an ad hoc manner using `puppet apply`. This requires an Influxdb admin token to be saved to `~/.influxdb_token` and for the dependencies to be installed locally, e.g. with:
influxdb_bucket String
Name of the InfluxDB bucket to configure and query. Defaults to 'puppet_data'
Default: 'puppet_data'
```
puppet module install puppetlabs/influxdb
puppet module install puppet/grafana
```
influxdb_org String
Name of the InfluxDB organization to configure. Defaults to 'puppetlabs'
Default: 'puppetlabs'
Next, save and configure the example file to `~/bucket_and_datasource.txt`:
influxdb_port Integer
Port used by the InfluxDB service. Defaults to the value of influxdb::port, or
8086 if unset
Default: 8086
```
# Example Puppet code to create an InfluxDB bucket and Grafana datasource.
# Intended to be used with `puppet apply`. Fill in these values as needed:
# <bucket_name>
# <my_org>
# <my_datasource>
class local_apply(
String $influxdb_token_file = lookup(influxdb::token_file, undef, undef, $facts['identity']['user'] ? {
'root' => '/root/.influxdb_token',
default => "/home/${facts['identity']['user']}/.influxdb_token"
})
){
$token = file($influxdb_token_file)
influxdb_bucket {'puppet_data':
ensure => present,
org => 'puppetlabs',
}
grafana_datasource {'sample_datasource':
grafana_user => 'admin',
grafana_password => 'admin',
grafana_url => "http://${facts['fqdn']}:3000",
type => 'influxdb',
# This must match the namevar of the influxdb_bucket resource
database => 'puppet_data',
url => "https://${facts['fqdn']}:8086",
access_mode => 'proxy',
is_default => false,
json_data => {
httpHeaderName1 => 'Authorization',
httpMode => 'GET',
tlsSkipVerify => true
},
secure_json_data => {
httpHeaderValue1 => "Token ${token}",
},
}
}
include local_apply
```
metrics_dir String
Path to the 'metrics' directory from a PE support script
And apply the code:
targets TargetSpec
The targets to run on.
```
puppet apply ~/bucket_and_datasource.txt
```
telegraf_process Enum['local', 'remote']
Default: 'remote'
For more information about the Grafana datasource, see the [puppet-grafana](https://github.com/voxpupuli/puppet-grafana/blob/master/REFERENCE.md#grafana_datasource) module.
telegraf_token String
Name of the token to retrieve from InfluxDB. Defaults to 'puppet telegraf token'
Default: 'puppet telegraf token'
### Run Telegraf to process and import the metrics
token Optional[String]
Default: undef
First, export your token described in the [requirements](#requirements).
```
export INFLUX_TOKEN=<token>
```
Then extract the archives, change to the appropriate directory as descriped in [metrics extraction](#metrics-extraction), and run Telegraf with the `--once` flag to import the metrics. This can be done all at once:
token_file String
Location on disk of an InfluxDB admin token.
This file is written to by the influxdb class during installation and read by
the type and providers,
as well Deferred functions in this module.
Default: '/root/.influxdb_token'
```
telegraf --once --debug --config ~/telegraf.conf --config-directory ~/telegraf.conf.d/
```

Or one service at a time, e.g. for Puppet server
```
telegraf --once --debug --config ~/telegraf.conf --config ~/telegraf.conf.d/puppetserver.conf
```

See [troubleshooting](https://github.com/puppetlabs/puppet_operational_dashboards#troubleshooting) in the `README` for information on troubleshooting Telegraf errors.

12 changes: 11 additions & 1 deletion CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,22 @@

All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](http://semver.org).

## [v1.4.0](https://github.com/puppetlabs/puppet_operational_dashboards/tree/v1.4.0) (2022-09-29)
## [v1.5.0](https://github.com/puppetlabs/puppet_operational_dashboards/tree/v1.5.0) (2022-10-11)

[Full Changelog](https://github.com/puppetlabs/puppet_operational_dashboards/compare/v1.4.0...v1.5.0)

### Added

- \(SUP-3688\) Plans to provision and import metrics [\#87](https://github.com/puppetlabs/puppet_operational_dashboards/pull/87) ([m0dular](https://github.com/m0dular))
- \(SUP-3565\) Support Telegraf archive install on EL [\#75](https://github.com/puppetlabs/puppet_operational_dashboards/pull/75) ([m0dular](https://github.com/m0dular))

## [v1.4.0](https://github.com/puppetlabs/puppet_operational_dashboards/tree/v1.4.0) (2022-09-30)

[Full Changelog](https://github.com/puppetlabs/puppet_operational_dashboards/compare/v1.3.0...v1.4.0)

### Added

- \(SUP-3675\) Remove -H flag from ss command [\#86](https://github.com/puppetlabs/puppet_operational_dashboards/pull/86) ([m0dular](https://github.com/m0dular))
- Pass use\_ssl param to InfluxDB resources [\#84](https://github.com/puppetlabs/puppet_operational_dashboards/pull/84) ([m0dular](https://github.com/m0dular))

## [v1.3.0](https://github.com/puppetlabs/puppet_operational_dashboards/tree/v1.3.0) (2022-09-19)
Expand Down
25 changes: 25 additions & 0 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
- [Description](#description)
- [Setup](#setup)
- [Prerequisites](#prerequisites)
- [Note on air-gapped environments](#note-on-air-gapped-environments)
- [Beginning with puppet_operational_dashboards](#beginning-with-puppet_operational_dashboards)
- [Installing on Puppet Enterprise](#installing-on-puppet-enterprise)
- [Installing on Puppet Open Source](#installing-on-puppet-open-source)
Expand Down Expand Up @@ -40,6 +41,30 @@ This module is a replacement for the [puppet_metrics_dashboard module](https://f

### Prerequisites

### Note on air-gapped environments

If you are applying either the `puppet_operational_dashboards` or `puppet_operational_dashboards::telegraf::agent` classes to a node that cannot access the internet, it is possible to install packages from an archive source located within the air gap.

For InfluxDB, set:

* [influxdb::manage_repo](https://forge.puppet.com/modules/puppetlabs/influxdb/reference#manage_repo) to `false`
* [influxdb::archive_source](https://forge.puppet.com/modules/puppetlabs/influxdb/reference#archive_source) to an internal URL containing the archive file for InfluxDB

For Grafana, set:

* [grafana::install_method](https://forge.puppet.com/modules/puppet/grafana/reference#install_method) to either `archive` or `package` depending on your preferred method.

Then, set either of the following depending on the installation method

* [grafana::package_source](https://forge.puppet.com/modules/puppet/grafana/reference#package_source)
* [grafana::archive_source](https://forge.puppet.com/modules/puppetlabs/influxdb/reference#archive_source)

For Telegraf, set:

* [puppet_operational_dashboards::telegraf::agent::manage_repo](https://forge.puppet.com/modules/puppetlabs/puppet_operational_dashboards/reference#manage_repo) to `false`
* [puppet_operational_dashboards::telegraf::agent::manage_archive](https://forge.puppet.com/modules/puppetlabs/puppet_operational_dashboards/reference#manage_archive) to `true`
* [puppet_operational_dashboards::telegraf::agent::archive_location](https://forge.puppet.com/modules/puppetlabs/puppet_operational_dashboards/reference#archive_location) to an internal URL containing the archive file for Telegraf

### Beginning with puppet_operational_dashboards

#### Installing on Puppet Enterprise
Expand Down
Loading