Puppet Configuration for LSB Infrastructure
===========================================

This repository contains (most of) the configuration for the LSB's
infrastructure.  We are moving to a devops model, where all server
configuration is done here instead of logging into a server and
munging configuration files.

We use Puppet (http://puppetlabs.com) as the configuration tool.  The
initial work done here was done with Puppet 2.6, and should remain
compatible with that version for the foreseeable future.

Documentation for specific services can be found in the documentation
directory of this repository.  This note will cover some general
topics related to the whole system.  If you find that something is not
adequately documented here, please contact the workgroup and let us
know what's not working for you.

Configuration is defined per-service in Puppet modules, and per-node
in the "nodes" directory.  Whenever possible, it's best to define
services as modules, rather than include all the gory details in the
per-node config.  This makes it easy to move services between nodes; a
quick "include 'foo'" creates the service on the node and gets it
working.

The configuration will, in general, reflect a SUSE bias, as the
initial node was a SLES 11 virtual machine.  However, we are trying to
isolate SUSE-specific things as much as possible, so we should be able
to accomodate nodes running other Linux distributions as well.  This
will obviously not be foolproof, so sticking with SUSE may be the best
bet for making things work well.  Although we're using SLES, OpenSUSE
should also work.

Please test configuration changes on a test node before pushing them
into production!

Please file bugs related to our configuration at the LSB bug tracker
(http://bugs.linuxbase.org), in the Infrastructure section.  There is
a DevOps component; this should be used only for bugs or request
specifically aimed at Puppet itself.  Bugs in other components that
happen to be configured in Puppet should be filed in that component,
not in DevOps.  Patches or bzr merge directives against this
repository are welcome to bugs in any Infrastructure component.

Adding a node to Puppet
-----------------------

Adding a node to this Puppet configuration is pretty much standard
Puppet, with only a few tweaks required for our local setup:

 - Check out the puppet-lsb repository from version control.

 - In puppet-lsb, create a node configuration file in manifest/nodes.
   You can use one of the other node configuration files as a model.
   Or, if you prefer, something like this is the minimum configuration
   required for a node:

   node 'my.host.name' { include puppet }

   The filename can be descriptive, but the actual node name in the
   file must correspond to the FQDN Puppet expects on the node.  You
   can find this hostname via this command:

   facter | grep fqdn

 - Once the node file is in place, use version control to add the new
   file, commit it, and push it.

 - On the node, install puppet, and run the following command as root:

   puppet agent --server puppet.linuxbase.org --waitforcert 60 --test

   Puppet should report that it is creating a new key and cert request
   and that the peer certificate won't be verified.  It should then
   wait, reporting every so often that it can't receive the
   certificate.

 - On the master, "puppet cert list" should report that a
   certification request exists for the node.  Run "puppet cert sign
   <hostname>" to sign the certificate.

 - On the node, the running puppet agent should report within 60
   seconds that it is caching the certificate, and should then
   retrieve and apply the node's configuration (if any) without
   further errors.

 - Since puppet resets things like sudo, it's possible to lose access
   to root after puppet has run.  We therefore recommend setting the
   password for the user 'lfadmin' and starting the puppet service
   manually once the puppet agent succeeds.

Running "puppet agent -t" on the node tests the connection, and also
can force the current configuration to apply immediately without
waiting for the normal agent update interval.

The puppet-secret modules
-------------------------

In a few cases, this configuration refers to modules that don't seem
to exist.  These are from the "puppet-secret" area, and contain
information that shouldn't be accessible to the public: passwords,
secret keys, and so on.

In all cases, puppet-secret contains modules which either expose
variables to Puppet, or install files.  Someone with experience in
Puppet should be able to recreate puppet-secret with nothing more than
the information contained here, and thus create an independent
working implementation.  Of course, the passwords, keys, etc. will be
different than on the live production copy.

It might be worth asking whether a "blank" public copy of
puppet-secrets could be provided.  The worry is that the blank copy
would get stale, and implementors would be forced to reverse-engineer
the missing bits from this repository anyway.  Suggestions for fixing
this problem are welcome.

Troubleshooting
---------------

There is a known issue with running a Puppet client with Ruby 1.9.2,
where the client refuses to complete the first-time connection with a
message like this:

err: Could not request certificate: SSL_connect returned=1 errno=0 \
  state=SSLv3 read server certificate B: certificate verify failed. \
  This is often because the time is out of sync on the server or client

(line reformatted for readability)

The problem has to do with an incompatible change in the way Ruby
1.9.2 validates certificates.  The problem and fix is discussed here:

http://urgetopunt.com/puppet/2011/09/14/puppet-ruby19.html

It consists of copying the master CA cert to the client manually, and
then linking it into OpenSSL's cert cache.  The file to copy should be
in /var/lib/puppet/ssl/certs/ca.pem if you're using distro-packaged
Puppet; that's both the location of the file on the master and its
destination on the client.  The symlink has to be a particular name:
[hash].0, where [hash] is calculated using this command:

openssl x509 -hash -noout -in /etc/puppet/ssl/certs/ca.pem

This symlink should live in /etc/ssl/certs on most Linux
distributions.  If you're not sure, the following command will tell
you where it should go:

openssl version -d

Once the file and symlink are in place on the client, retry the
initial puppet connection.  It should work.

Jeff Licquia
licquia@linuxfoundation.org
2014-04-23