4 changes: 4 additions & 0 deletions .vscode/settings.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"python.linting.pylintEnabled": true,
"python.linting.enabled": true
}
13 changes: 13 additions & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,19 @@

All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](http://semver.org).

## [v3.6.0](https://github.com/puppetlabs/puppetlabs-peadm/tree/v3.6.0) (2022-05-04)

[Full Changelog](https://github.com/puppetlabs/puppetlabs-peadm/compare/v3.5.0...v3.6.0)

### Added

- Solarch 581 [\#235](https://github.com/puppetlabs/puppetlabs-peadm/pull/235) ([davidsandilands](https://github.com/davidsandilands))

### Fixed

- Add read map so puppetdb can start on new compiler [\#252](https://github.com/puppetlabs/puppetlabs-peadm/pull/252) ([ody](https://github.com/ody))
- \(SOLARCH-434\) Procedure for recovering PSQL [\#243](https://github.com/puppetlabs/puppetlabs-peadm/pull/243) ([ody](https://github.com/ody))

## [v3.5.0](https://github.com/puppetlabs/puppetlabs-peadm/tree/v3.5.0) (2022-04-13)

[Full Changelog](https://github.com/puppetlabs/puppetlabs-peadm/compare/v3.4.0...v3.5.0)
Expand Down
88 changes: 37 additions & 51 deletions REFERENCE.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -35,12 +35,14 @@
* [`peadm::node_manager_yaml_location`](#peadmnode_manager_yaml_location)
* [`peadm::oid`](#peadmoid)
* [`peadm::plan_step`](#peadmplan_step)
* [`peadm::recovery_opts_default`](#peadmrecovery_opts_default)
* [`peadm::wait_until_service_ready`](#peadmwait_until_service_ready): A convenience function to help remember port numbers for services and handle running the wait_until_service_ready task

### Data types

* [`Peadm::Pe_version`](#peadmpe_version)
* [`Peadm::Pem`](#peadmpem)
* [`Peadm::Recovery_opts`](#peadmrecovery_opts)
* [`Peadm::SingleTargetSpec`](#peadmsingletargetspec): A SingleTargetSpec represents any String, Target or single-element array of one or the other that can be passed to get_targets() to return an

### Tasks
Expand Down Expand Up @@ -725,6 +727,18 @@ Data type: `Callable`



### <a name="peadmrecovery_opts_default"></a>`peadm::recovery_opts_default`

Type: Puppet Language

The peadm::recovery_opts_default function.

#### `peadm::recovery_opts_default()`

The peadm::recovery_opts_default function.

Returns: `Any`

### <a name="peadmwait_until_service_ready"></a>`peadm::wait_until_service_ready`

Type: Puppet Language
Expand Down Expand Up @@ -773,6 +787,23 @@ Alias of
Pattern[/^-----BEGIN/]
```

### <a name="peadmrecovery_opts"></a>`Peadm::Recovery_opts`

The Peadm::Recovery_opts data type.

Alias of

```puppet
Struct[{
'orchestrator' => Optional[Boolean],
'puppetdb' => Optional[Boolean],
'rbac' => Optional[Boolean],
'activity' => Optional[Boolean],
'ca' => Optional[Boolean[false]],
'classifier' => Optional[Boolean],
}]
```

### <a name="peadmsingletargetspec"></a>`Peadm::SingleTargetSpec`

A SingleTargetSpec represents any String, Target or single-element array of
Expand Down Expand Up @@ -1290,68 +1321,23 @@ This plan can backup data as outlined at insert doc

The following parameters are available in the `peadm::backup` plan:

* [`primary_host`](#primary_host)
* [`backup_orchestrator`](#backup_orchestrator)
* [`backup_rbac`](#backup_rbac)
* [`backup_activity`](#backup_activity)
* [`backup_ca_ssl`](#backup_ca_ssl)
* [`backup_puppetdb`](#backup_puppetdb)
* [`backup_classification`](#backup_classification)
* [`targets`](#targets)
* [`backup`](#backup)
* [`output_directory`](#output_directory)

##### <a name="primary_host"></a>`primary_host`
##### <a name="targets"></a>`targets`

Data type: `Peadm::SingleTargetSpec`



##### <a name="backup_orchestrator"></a>`backup_orchestrator`

Data type: `Boolean`



Default value: ``true``

##### <a name="backup_rbac"></a>`backup_rbac`

Data type: `Boolean`



Default value: ``true``

##### <a name="backup_activity"></a>`backup_activity`

Data type: `Boolean`



Default value: ``true``

##### <a name="backup_ca_ssl"></a>`backup_ca_ssl`

Data type: `Boolean`



Default value: ``true``

##### <a name="backup_puppetdb"></a>`backup_puppetdb`

Data type: `Boolean`


##### <a name="backup"></a>`backup`

Default value: ``false``
Data type: `Peadm::Recovery_opts`

##### <a name="backup_classification"></a>`backup_classification`

Data type: `Boolean`



Default value: ``true``
Default value: `{}`

##### <a name="output_directory"></a>`output_directory`

Expand Down
37 changes: 37 additions & 0 deletions documentation/automated_recovery.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# Recovery procedures

These instructions provide automated procedures for recovering from select failures of PE components which are managed by PEADM.

Additional manual procedures are documented in [recovery.md](recovery.md)

## Replace failed PE-PostgreSQL server (A or B side)

The procedure for replacing a failed PE-PostgreSQL server is the same regardless of which PE-PostgreSQL server is missing or if the name of the PE-PostgrSQL server is the same or different. This procedure uses the following placeholder references.

* _\<replacement-postgres-server-fqdn\>_ - The FQDN and certname of the new server being brought in to replace the failed PE-PostgreSQL server
* _\<working-postgres-server-fqdn\>_ - The FQDN and certname of the still-working PE-PostgreSQL server
* _\<failed-postgres-server-fqdn\>_ - The FQDN and certname of the failed PE-PostgreSQL server
* _\<primary-server-fqdn\>_ - The FQDN and certname of the primary Puppet server
* _\<replica-server-fqdn\>_ - The FQDN and certname of the replica Puppet server

Procedure:

1. Stop `puppet.service` on Puppet server primary and replica

bolt task run service name=puppet.service action=stop --targets <primary-server-fqdn>,<replica-server-fqdn>

2. Temporarily set both primary and replica server nodes so that they use the remaining healthy PE-PostgreSQL server

bolt plan run peadm::util::update_db_setting --target <primary-server-fqdn>,<replica-server-fqdn> primary_postgresql_host=<working-postgres-server-fqdn>

3. Restart `pe-puppetdb.service` on Puppet server primary and replica

bolt task run service name=pe-puppetdb.service action=restart --targets <primary-server-fqdn>,<replica-server-fqdn>

4. Purge failed PE-PostgreSQL node from PuppetDB

bolt command run "/opt/puppetlabs/bin/puppet node purge <failed-postgres-server-fqdn>" --targets <primary-server-fqdn>

5. Run `peadm::add_database` plan to deploy replacement PE-PostgreSQL server

bolt plan run peadm::add_database -t <replacement-postgres-server-fqdn> primary_host=<primary-server-fqdn>
10 changes: 10 additions & 0 deletions functions/recovery_opts_default.pp
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
function peadm::recovery_opts_default () {
{
'orchestrator' => true,
'puppetdb' => true,
'rbac' => true,
'activity' => true,
'ca' => false,
'classifier' => true,
}
}
2 changes: 1 addition & 1 deletion metadata.json
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "puppetlabs-peadm",
"version": "3.5.0",
"version": "3.6.0",
"author": "puppetlabs",
"summary": "Bolt plans used to deploy an at-scale Puppet Enterprise architecture",
"license": "Apache-2.0",
Expand Down
4 changes: 4 additions & 0 deletions plans/add_compiler.pp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,10 @@
path => '/opt/puppetlabs/server/data/postgresql/11/data/pg_ident.conf',
line => "pe-puppetdb-pe-puppetdb-migrator-map ${compiler_target.peadm::certname()} pe-puppetdb-migrator",
}
file_line { 'pe-puppetdb-pe-puppetdb-read-map':
path => '/opt/puppetlabs/server/data/postgresql/11/data/pg_ident.conf',
line => "pe-puppetdb-pe-puppetdb-read-map ${compiler_target.peadm::certname()} pe-puppetdb-read",
}
}

# Reload pe-postgresql.service
Expand Down
127 changes: 84 additions & 43 deletions plans/backup.pp
View file
Original file line number Diff line number Diff line change
@@ -1,77 +1,118 @@
# @summary Backup the core user settings for puppet infrastructure
#
# This plan can backup data as outlined at insert doc
# This plan can backup data as outlined at insert doc
#
plan peadm::backup (
Peadm::SingleTargetSpec $primary_host,
# This plan should be run on the primary server
Peadm::SingleTargetSpec $targets,

# Which data to backup
Boolean $backup_orchestrator = true,
Boolean $backup_rbac = true,
Boolean $backup_activity = true,
Boolean $backup_ca_ssl = true,
Boolean $backup_puppetdb = false,
Boolean $backup_classification = true,
String $output_directory = '/tmp',
Peadm::Recovery_opts $backup = {},

# Where to put the backup folder
String $output_directory = '/tmp',
) {
peadm::assert_supported_bolt_version()
$cluster = run_task('peadm::get_peadm_config', $primary_host).first

$recovery_opts = (peadm::recovery_opts_default() + $backup)
$cluster = run_task('peadm::get_peadm_config', $targets).first.value
$arch = peadm::assert_supported_architecture(
$primary_host,
$cluster['replica_host'],
$cluster['primary_postgresql_host'],
$cluster['replica_postgresql_host'],
$cluster['compiler_hosts'],
getvar('cluster.params.primary_host'),
getvar('cluster.params.replica_host'),
getvar('cluster.params.primary_postgresql_host'),
getvar('cluster.params.replica_postgresql_host'),
getvar('cluster.params.compiler_hosts'),
)

$timestamp = Timestamp.new().strftime('%F_%T')
$timestamp = Timestamp.new().strftime('%Y-%m-%dT%H%M%SZ')
$backup_directory = "${output_directory}/pe-backup-${timestamp}"

# Create backup folder
apply($primary_host){
$primary_target = getvar('cluster.params.primary_host')
$puppetdb_postgresql_target = getvar('cluster.params.primary_postgresql_host') ? {
undef => getvar('cluster.params.primary_host'),
default => getvar('cluster.params.primary_postgresql_host'),
}

$backup_databases = {
'orchestrator' => $primary_target,
'activity' => $primary_target,
'rbac' => $primary_target,
'puppetdb' => $puppetdb_postgresql_target,
}.filter |$key,$_| {
$recovery_opts[$key] == true
}

# Create backup folders
apply($primary_target) {
file { $backup_directory :
ensure => 'directory',
owner => 'root',
group => 'pe-postgres',
mode => '0770'
group => 'root',
mode => '0700'
}
}

# Create an array of the names of databases and whether they have to be backed up to use in a lambda later
$database_to_backup = [ $backup_orchestrator, $backup_activity, $backup_rbac, $backup_puppetdb]
$database_names = [ 'pe-orchestrator' , 'pe-activity' , 'pe-rbac' , 'pe-puppetdb' ]
# Create a subdir for each backup type selected
$recovery_opts.filter |$_,$val| { $val == true }.each |$dir,$_| {
file { "${backup_directory}/${dir}":
ensure => 'directory',
owner => 'root',
group => 'root',
mode => '0700'
}
}
}

if $backup_classification {
if getvar('recovery_opts.classifier') {
out::message('# Backing up classification')
run_task('peadm::backup_classification', $primary_host,
directory => $backup_directory,
run_task('peadm::backup_classification', $primary_target,
directory => "${backup_directory}/classifier",
)
}

if $backup_ca_ssl {
if getvar('recovery_opts.ca') {
out::message('# Backing up ca and ssl certificates')
run_command("/opt/puppetlabs/bin/puppet-backup create --dir=${backup_directory} --scope=certs", $primary_host)
run_command(@("CMD"), $primary_target)
/opt/puppetlabs/bin/puppet-backup create --dir=${shellquote($backup_directory)}/ca --scope=certs
| CMD
}

# Check if /etc/puppetlabs/console-services/conf.d/secrets/keys.json exists and if so back it up
out::message('# Backing up ldap secret key if it exists')
run_command("test -f /etc/puppetlabs/console-services/conf.d/secrets/keys.json && cp -rp /etc/puppetlabs/console-services/conf.d/secrets/keys.json ${backup_directory} || echo secret ldap key doesnt exist" , $primary_host) # lint:ignore:140chars
if getvar('recovery_opts.rbac') {
out::message('# Backing up ldap secret key if it exists')
run_command(@("CMD"/L), $primary_target)
test -f /etc/puppetlabs/console-services/conf.d/secrets/keys.json \
&& cp -rp /etc/puppetlabs/console-services/conf.d/secrets ${shellquote($backup_directory)}/rbac/ \
|| echo secret ldap key doesnt exist
| CMD
}

# IF backing up orchestrator back up the secrets too /etc/puppetlabs/orchestration-services/conf.d/secrets/
if $backup_orchestrator {
if getvar('recovery_opts.orchestrator') {
out::message('# Backing up orchestrator secret keys')
run_command("cp -rp /etc/puppetlabs/orchestration-services/conf.d/secrets ${backup_directory}/", $primary_host)
run_command(@("CMD"), $primary_target)
cp -rp /etc/puppetlabs/orchestration-services/conf.d/secrets ${shellquote($backup_directory)}/orchestrator/
| CMD
}

$database_to_backup.each |Integer $index, Boolean $value | {
if $value {
out::message("# Backing up database ${database_names[$index]}")
# If the primary postgresql host is set then pe-puppetdb needs to be remotely backed up to primary.
if $database_names[$index] == 'pe-puppetdb' and $cluster['primary_postgresql_host'] {
run_command("sudo -u pe-puppetdb /opt/puppetlabs/server/bin/pg_dump \"sslmode=verify-ca host=${cluster['primary_postgresql_host']} sslcert=/etc/puppetlabs/puppetdb/ssl/${primary_host}.cert.pem sslkey=/etc/puppetlabs/puppetdb/ssl/${primary_host}.private_key.pem sslrootcert=/etc/puppetlabs/puppet/ssl/certs/ca.pem dbname=pe-puppetdb\" -f /tmp/puppetdb_$(date +%F_%T).bin" , $primary_host) # lint:ignore:140chars
} else {
run_command("sudo -u pe-postgres /opt/puppetlabs/server/bin/pg_dump -Fc \"${database_names[$index]}\" -f \"${backup_directory}/${database_names[$index]}_$(date +%F_%T).bin\"" , $primary_host) # lint:ignore:140chars
}
}
$backup_databases.each |$name,$database_target| {
run_command(@("CMD"/L), $primary_target)
/opt/puppetlabs/server/bin/pg_dump -Fd -Z3 -j4 \
-f ${shellquote($backup_directory)}/${shellquote($name)}/pe-${shellquote($name)}.dump.d \
"sslmode=verify-ca \
host=${shellquote($database_target.peadm::certname())} \
user=pe-${shellquote($name)} \
sslcert=/etc/puppetlabs/puppetdb/ssl/${shellquote($primary_target.peadm::certname())}.cert.pem \
sslkey=/etc/puppetlabs/puppetdb/ssl/${shellquote($primary_target.peadm::certname())}.private_key.pem \
sslrootcert=/etc/puppetlabs/puppet/ssl/certs/ca.pem \
dbname=pe-${shellquote($name)}"
| CMD
}

run_command(@("CMD"/L), $primary_target)
umask 0077 \
&& tar -czf ${shellquote($backup_directory)}.tar.gz ${shellquote($backup_directory)} \
&& rm -rf ${shellquote($backup_directory)}
| CMD

return({'path' => "${backup_directory}.tar.gz"})
}
Loading