1 change: 1 addition & 0 deletions .fixtures.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
fixtures:
forge_modules:
ruby_task_helper: "puppetlabs/ruby_task_helper"
service: "puppetlabs/service"
repositories:
facts: 'https://github.com/puppetlabs/puppetlabs-facts.git'
puppet_agent: 'https://github.com/puppetlabs/puppetlabs-puppet_agent.git'
Expand Down
22 changes: 22 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,25 @@
# PEADM module

## 2.2.0
### Summary

Reliability fixes for 2019.8.1, README updates, and simpification of the convert plan. New parameters added for `internal_compiler_a_pool_address` and `internal_compiler_b_pool_address` to configure lb addresses for each half of the compiler pool, so that this configuration does not need to be re-applied after upgrades.

### Features

- Added parameters to configure compiler pool addresses for the A and B availability groups. These are used in large and extra large architectures.
- Add basic informational messages to upgrade plan output, to communicate when different stages of the upgrade begin.

### Bugfixes

- Fixed GH-118, wherein a compiler would unnecessarily send duplicate work to an extra configured PuppetDB endpoint.
- Puppet infra upgrade operations now always wait until target nodes are connected before attempting an operation

### Improvements

- Provide a useful overview of the module in the README so that readers can quickly gain a sense of how the module is used, what it affects, and what it does not affect.
- Eliminate `configure_node_groups` parameter to peadm::convert. Perform the correct action(s) automatically.

## Release 2.1.1
### Summary

Expand Down
55 changes: 43 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,38 +1,69 @@
# Puppet Enterprise (pe) Administration (adm) Module

This Puppet module contains Bolt plans used to deploy and manage Puppet Enterprise infrastructure. Plans are provided to automate common lifecycle activities, in order to increase velocity and reduce the possibility of human error incurred by manually performing these activities.
This Puppet module contains Bolt plans used to deploy and manage Puppet Enterprise infrastructure. Plans are provided to automate common lifecycle activities in order to increase velocity and reduce the possibility of human error incurred by manually performing these activities.

The peadm module is able to deploy and manage Puppet Enterprise 2019.x Standard, Large, and Extra Large architectures.

#### Table of Contents

1. [Expectations](#expectations)
1. [Overview](#overview)
* [What peadm affects](#what-peadm-affects)
* [What peadm does not affect](#what-peadm-does-not-affect)
* [Requirements](#requirements)
3. [Usage](#usage)
4. [Reference](#reference)
5. [Getting Help](#getting-help)

## Expectations

The peadm module is intended to be used only by Puppet Enterprise customers actively working with and being guided by Puppet Customer Success teams—specifically, the Professional Services and Solutions Architecture teams. Independent use is not recommended for production environments without a comprehensive understanding of the peadm module.

## Documentation
## Overview

The normal usage pattern for peadm is as follows.

1. Users set up a Bolt host from which they can run peadm plans. The Bolt host can be any machine that has ssh access to all of the PE nodes.
2. Users run the `peadm::provision` plan to bootstrap a new PE deployment. Depending on the architecture chosen, peadm may create some node groups in the classifier to set parameters on the built-in `puppet_enterprise` module, tuning it for large or extra large architectures.
3. Users use and operate their PE deployment as normal. The peadm module is not used again until the next upgrade.
4. When it is time to upgrade, users run the `peadm::upgrade` plan from their Bolt host to accelerate and aid in the upgrade process.

### What peadm affects

* The `peadm::provision` plan adds a number of custom OID trusted facts to the certificates of PE infrastructure nodes as it deploys them. These trusted facts are later used by the plans to quickly and correctly identify nodes in particular roles.
* Up to four node groups may be created to help configure `puppet_enterprise` class parameters for PE infrastructure roles. The most notable configuration is the designation of compilers as being either "A" or "B" nodes for availability.

### What peadm does not affect

See this README file and any documents in the [documentation](documentation) directory.
* The peadm module is not required to exist or be present outside of the point(s) in time it is used to create a new PE deployment, or upgrade an existing deployment. No new Puppet classes or other persistent content not provided out-of-box by PE itself is applied to PE infrastructure nodes by the peadm module.
* Having used the peadm module to provision or to upgrade a PE deployment is not known to affect or curtail the ability to use any normal, documented PE procedures, e.g. failover to a replica, or manual upgrade of a deployment.

Plans:
### Requirements

* Puppet Enterprise 2019.8.1 or newer (tested with PE 2019.8.1)
* Bolt 2.27.0 or newer (tested with Bolt 2.27.0)
* EL 7, EL 8, Ubuntu 18.04, or Ubuntu 20.04
* Classifier Data enabled. This PE feature is enabled by default on new installs, but can be disabled by users if they remove the relevant configuration from their global hiera.yaml file. See the [PE docs](https://puppet.com/docs/pe/latest/config_console.html#task-5039) for more information.

## Usage

Follow the links below to usage instructions for each peadm plan.

* [Provision](documentation/provision.md)
* [Upgrade](documentation/upgrade.md)
* [Convert](documentation/convert.md)
* [Status](documentation/status.md)

Reference:
## Reference

Additional documentation and information pertaining to various aspects or elements of peadm.

* [PE Architecture Documentation](https://puppet.com/docs/pe/latest/choosing_an_architecture.html)
* [Classification](documentation/classification.md)
* [Architectures](documentation/architectures.md)
* [Testing](documentation/pre_post_checks.md)
* [Docker Based Examples](documentation/docker_examples.md)

## Requirements

Normally, if you are able to use peadm to set up an infrastructure and Puppet agent runs are all working, chances are you met all the requirements and don't have to worry. Sometimes Some notable requirements are highlighted below which may accidentally be adjusted by users, but which architectures deployed by this module rely on. These configuration requirements need to be maintained for the infrastructure to operate correctly.

* Classifier Data needs to be enabled. This feature is enabled by default on new installs, but can be disabled by users if they remove the relevant configuration from their global hiera.yaml file. See the [PE docs](https://puppet.com/docs/pe/latest/config_console.html#task-5039) for more information.

## Getting Help

To get help with issues concerning this module, please make use of [issues](https://github.com/puppetlabs/puppetlabs-peadm/issues) in the project on GitHub.
To get help with issues concerning this module, please make use of [issues](https://github.com/puppetlabs/puppetlabs-peadm/issues) in the project on GitHub.
1 change: 1 addition & 0 deletions functions/oid.pp
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ function peadm::oid (
'peadm_availability_group': { '1.3.6.1.4.1.34380.1.1.9813' }
'pp_application': { '1.3.6.1.4.1.34380.1.1.8' }
'pp_cluster': { '1.3.6.1.4.1.34380.1.1.16' }
'pp_role': { '1.3.6.1.4.1.34380.1.1.13' }
'pp_auth_role': { '1.3.6.1.4.1.34380.1.3.13' }
default: { fail("No peadm OID for ${short_name}") }
}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# @summary Defines configuration needed for converting PE 2018
#
class peadm::setup::convert_pe2018 {
class peadm::setup::convert_pre20197 {

# This is needed so that compiler certs can be signed. It's included by
# default in 2019.7 and newer, but isn't present in 2018.1. It would be
Expand Down
38 changes: 32 additions & 6 deletions manifests/setup/node_manager.pp
Original file line number Diff line number Diff line change
Expand Up @@ -9,14 +9,30 @@
# environments => ["production", "staging", "development"],
# }'
#
# @param compiler_pool_address
# The service address used by agents to connect to compilers, or the Puppet
# service. Typically this is a load balancer.
# @param internal_compiler_a_pool_address
# A load balancer address directing traffic to any of the "A" pool
# compilers. This is used for DR/HA configuration in large and extra large
# architectures.
# @param internal_compiler_b_pool_address
# A load balancer address directing traffic to any of the "B" pool
# compilers. This is used for DR/HA configuration in large and extra large
# architectures.
#
class peadm::setup::node_manager (
# Common
# Standard
String[1] $master_host,
Optional[String[1]] $compiler_pool_address = undef,

# High Availability
Optional[String[1]] $master_replica_host = undef,

# Common
Optional[String[1]] $compiler_pool_address = undef,
Optional[String[1]] $internal_compiler_a_pool_address = $master_host,
Optional[String[1]] $internal_compiler_b_pool_address = $master_replica_host,

# For the next two parameters, the default values are appropriate when
# deploying Standard or Large architectures. These values only need to be
# specified differently when deploying an Extra Large architecture.
Expand Down Expand Up @@ -92,7 +108,6 @@
node_group { 'PE Compiler Group A':
ensure => 'present',
parent => 'PE Compiler',
data => { },
rule => ['and',
['=', ['trusted', 'extensions', 'pp_auth_role'], 'pe_compiler'],
['=', ['trusted', 'extensions', peadm::oid('peadm_availability_group')], 'A'],
Expand All @@ -102,10 +117,16 @@
'database_host' => $puppetdb_database_host,
},
'puppet_enterprise::profile::master' => {
'puppetdb_host' => ['${trusted[\'certname\']}', $master_replica_host].filter |$_| { $_ }, # lint:ignore:single_quote_string_with_variables
'puppetdb_host' => ['${trusted[\'certname\']}', $internal_compiler_b_pool_address].filter |$_| { $_ }, # lint:ignore:single_quote_string_with_variables
'puppetdb_port' => [8081],
}
},
data => {
# Workaround for GH-118
'puppet_enterprise::profile::master::puppetdb' => {
'ha_enabled_replicas' => [ ],
},
},
}

# Create the replica and B groups if a replica master and database host are
Expand Down Expand Up @@ -144,7 +165,6 @@
node_group { 'PE Compiler Group B':
ensure => 'present',
parent => 'PE Compiler',
data => { },
rule => ['and',
['=', ['trusted', 'extensions', 'pp_auth_role'], 'pe_compiler'],
['=', ['trusted', 'extensions', peadm::oid('peadm_availability_group')], 'B'],
Expand All @@ -154,10 +174,16 @@
'database_host' => $puppetdb_database_replica_host,
},
'puppet_enterprise::profile::master' => {
'puppetdb_host' => ['${trusted[\'certname\']}', $master_host], # lint:ignore:single_quote_string_with_variables
'puppetdb_host' => ['${trusted[\'certname\']}', $internal_compiler_a_pool_address], # lint:ignore:single_quote_string_with_variables
'puppetdb_port' => [8081],
}
},
data => {
# Workaround for GH-118
'puppet_enterprise::profile::master::puppetdb' => {
'ha_enabled_replicas' => [ ],
},
},
}
}

Expand Down
6 changes: 5 additions & 1 deletion metadata.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "puppetlabs-peadm",
"version": "2.1.1",
"version": "2.2.0",
"author": "puppetlabs",
"summary": "Bolt plans used to deploy an at-scale Puppet Enterprise architecture",
"license": "Apache-2.0",
Expand All @@ -27,6 +27,10 @@
{
"name": "puppet/format",
"version_requirement": ">= 0.1.0 < 1.0.0"
},
{
"name": "puppetlabs/service",
"version_requirement": ">= 1.3.0 < 2.0.0"
}
],
"operatingsystem_support": [
Expand Down
28 changes: 22 additions & 6 deletions plans/action/configure.pp
Original file line number Diff line number Diff line change
@@ -1,5 +1,17 @@
# @summary Configure first-time classification and HA setup
#
# @param compiler_pool_address
# The service address used by agents to connect to compilers, or the Puppet
# service. Typically this is a load balancer.
# @param internal_compiler_a_pool_address
# A load balancer address directing traffic to any of the "A" pool
# compilers. This is used for DR/HA configuration in large and extra large
# architectures.
# @param internal_compiler_b_pool_address
# A load balancer address directing traffic to any of the "B" pool
# compilers. This is used for DR/HA configuration in large and extra large
# architectures.
#
plan peadm::action::configure (
# Standard
Peadm::SingleTargetSpec $master_host,
Expand All @@ -14,6 +26,8 @@

# Common Configuration
String $compiler_pool_address = $master_host,
Optional[String] $internal_compiler_a_pool_address = undef,
Optional[String] $compiler_pool_b_address = undef,
Optional[String] $token_file = undef,
Optional[String] $deploy_environment = undef,

Expand Down Expand Up @@ -64,12 +78,14 @@
}

class { 'peadm::setup::node_manager':
master_host => $master_target.peadm::target_name(),
master_replica_host => $master_replica_target.peadm::target_name(),
puppetdb_database_host => $puppetdb_database_target.peadm::target_name(),
puppetdb_database_replica_host => $puppetdb_database_replica_target.peadm::target_name(),
compiler_pool_address => $compiler_pool_address,
require => Class['peadm::setup::node_manager_yaml'],
master_host => $master_target.peadm::target_name(),
master_replica_host => $master_replica_target.peadm::target_name(),
puppetdb_database_host => $puppetdb_database_target.peadm::target_name(),
puppetdb_database_replica_host => $puppetdb_database_replica_target.peadm::target_name(),
compiler_pool_address => $compiler_pool_address,
internal_compiler_a_pool_address => $internal_compiler_a_pool_address,
internal_compiler_b_pool_address => $internal_compiler_b_pool_address,
require => Class['peadm::setup::node_manager_yaml'],
}
}

Expand Down
Loading