Puppet Module Adoption

A tool for automating a workflow to adopt Puppet modules from the Puppet Forge without the need to write Puppet code.

##Usage

puppet adopter module <module name>

Example Run

[root@master ~]# puppet adopter module puppetlabs-ntp
Notice: Preparing to run experiment for module 'puppetlabs-ntp'
Notice: Installing Modules...
Notice: Preparing to install into /etc/puppetlabs/code/environments/production/modules ...
Notice: Downloading from https://forgeapi.puppetlabs.com ...
Notice: Installing -- do not interrupt ...
Could not find group with the name Adopter Experiment: ntp
Notice: Creating new group for experiment...
Notice: Check classification for "Adopter Experiment: ntp" in the Enterprise Console before continuing
Notice: Navigate a browser to https://master.puppetlabs.vm/#/node_groups/groups/f1fd495d-f570-4a3e-9336-b6bca10367fe
When you are ready, press enter to continue:
Notice: Starting Puppet Agent runs on experiment population
Nodes Complete: |==============================================================|
Notice: Puppet Agent runs completed


Total Variations Discovered: 2

Variation 1
    Total Events: 3
    Total Nodes:  1
---

Event - Service[ntp]
    Proporty:  ensure
    Old Value: stopped
    NewValue:  running
    Message:  current_value stopped, should be running (noop)
---
Event - File[/etc/ntp.conf]
    Proporty:  ensure
    Old Value: absent
    NewValue:  file
    Message:  current_value absent, should be file (noop)
---
Event - Package[ntp]
    Proporty:  ensure
    Old Value: purged
    NewValue:  present
    Message:  current_value purged, should be present (noop)
---
Nodes:
    master.puppetlabs.vm
-----------------END VARIATION 1-------------
Variation 2
    Total Events: 2
    Total Nodes:  2
---

Event - Service[ntp]
    Proporty:  ensure
    Old Value: stopped
    NewValue:  running
    Message:  current_value stopped, should be running (noop)
---
Event - File[/etc/ntp.conf]
    Proporty:  content
    Old Value: {md5}dc9e5754ad2bb6f6c32b954c04431d0a
    NewValue:  {md5}1f44e40bd99abd89f0a209e823285332
    Message:  current_value {md5}dc9e5754ad2bb6f6c32b954c04431d0a, should be {md5}1f44e40bd99abd89f0a209e823285332 (noop)
---
Nodes:
    agent1.puppetlabs.vm
    agent2.puppetlabs.vm
-----------------END VARIATION 2-------------
[root@master ~]#

##Workflow

The workflow explained below is a high-level description of the workflow that has been automated by this tool.

1. Select a module from the Puppet Forge that we would like to introduce into Puppet Enterprise to manage some component in the infrastructure.
2. Install the selected module into Puppet Enterprise
3. Establish a group in the Enterprise Console. Use this group to select the nodes that will be evaluated to managed by the module. Apply the necessary features of module and set any parameters needed.
4. Run the Puppet Agent across the selected population in "noop" mode.
5. Collect the "events" that have been generated by the Puppet Agent runs.
6. Correlate the events to determine groups of nodes that would have similar changes performed to reach desired end state.
7. Review the groups one-by-one and determine if there is a risk to applying Puppet in enforce mode.
8. Resolve the potential changes of the machines, either manually or via Puppet, until it is safe to run Puppet in enforce mode.
9. Repeat!

##How It Works

This tool will ultimately automate almost all of the steps in the process above. If used correctly, steps 2, most of 3, 4,5,6, and 7 will by handled by this tool.

When this tool is run, the following things will be done:

1. Install the Puppet module in question from the Puppet Forge if needed.
2. Check for existing Classifier group, if not create a new group.

• If the group needs to be created, the tool will create a group with a very broad base. The group will attempt to classify the default Puppet Class from the module if present, It will create a classification rule matching the entire population of nodes in Puppet Enterprise. Finally it will set a variable that will be caught by the "noop_from_console" function that will set the classes in this group to "noop" mode.

3. The tool will confirm that the group is ready, and wait for the user to confirm.
4. Upon confirmation, the tool will use the PXP system to execute Puppet Agent runs on each of the nodes in the group.
5. The Puppet agent runs will either complete or a timer will expire.
6. Once the Puppet runs are complete, the events are collected and processed to generate one or more Variation groups.
7. Results are displayed for the user to review.

##Setup

The steps below are initial installation steps for this tool while it is still in a development phase.

###Compatibility:

• Puppet Enterprise 2016.1.0 or up
• Runs on monolithic Installation of PE Only
• Must be installed and run from the Master of Masters (MoM)

Install adopt module

Eventually this will be hosted on the forge, so you can install just with:

puppet module install puppetlabs-adopt

In the mean time you'll need to git clone:

cd /etc/puppetlabs/code/modules`
git clone https://github.com/trlinkin/puppet-adopt.git adopt`

Create a configuration file

Create a configuration file in /etc/puppetlabs/puppet/adopter.yaml to specify the servers that host the PuppetDB and the Node Classifier in a split install. If this file is not present, a monolithic installed would be assumed and PDB and NC hosts will default to the master.

---
pdb_host: "puppetdb.puppet.vm"
nc_host: "console.puppet.vm"

Classification Configuration

1. Create a new node group called "Master Adopter"
2. Add the "adopt" class
3. Pin to the Puppet master:

Manual Install:

A Number of Gems will need to be installed into Puppet Enterprise for this tool to function. This is automated in the adopt module, but if you choose to do a manual install, these are the steps to follow:

Gems installed into PE Agent Ruby

• Puppet Classify
  • /opt/puppetlabs/puppet/bin/gem install puppetclassify
• PuppetDB Ruby Client
  • /opt/puppetlabs/puppet/bin/gem install puppetdb-ruby
• PCP Ruby Client
  • This Gem will need to build "EventMachine" and this will require the master to have ability to build software.
  • /opt/puppetlabs/puppet/bin/gem install pcp-client
• Inquirer
  • /opt/puppetlabs/puppet/bin/gem install inquirer
• Progress Bar
  • /opt/puppetlabs/puppet/bin/gem install ruby-progressbar

Gems installed into the PuppetServer JRuby

• Puppet Classify
  • /opt/puppetlabs/bin/puppetserver gem install puppetclassify
  • Restart the pe-puppetserver service after installation

###Prepare Puppet Master

1. Install trlinkin/noop module
   1. cd /etc/puppetlabs/code/modules
   2. git clone https://github.com/trlinkin/trlinkin-noop.git noop
   3. cd noop
   4. git checkout noop_from_console
2. Add "noop" function to the default manifest
   1. Open /etc/puppetlabs/code/environments/production/manifests/site.pp for editing
   2. Add the following code to the end of this file: noop_from_console('noop')

Add new rule to PCP Broker

The PCP Broker will need to allow the tool to orchestrate Puppet Agent runs on the nodes in our test population. We will need to configure the Broker to allow these messages to be sent.

• Add a new rule to the Broker authorization configuration: /etc/puppetlabs/orchestration-services/conf.d/authorization.conf

      {
          "allow" : [
              "pe-internal-dashboard",
              "pe-internal-orchestrator",
              "master.puppetlabs.vm"
          ],
          "match-request" : {
              "path" : "/pcp-broker/send",
              "query-params" : {
                  "message_type" : [
                      "http://puppetlabs.com/rpc_non_blocking_request",
                      "http://puppetlabs.com/rpc_blocking_request"
                  ]
              },
              "type" : "path"
          },
          "name" : "adopter pxp commands",
          "sort-order" : 300
      }
  ,

• Restart pe-orchestration-services service for the rule to take effect