Switch branches/tags
Nothing to show
Find file History
Pull request Compare This branch is 12 commits behind jordansissel:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
..
Failed to load latest commit information.
manifests
modules/example/manifests
README.rdoc
runashost.sh

README.rdoc

Exported Resources that expire

Why?

When a host is removed from your infrastructure or otherwise ceases to function (intentionally) as a particular role, it's useful to notify the rest of your infrastructure - your monitoring, for example. In a pure-puppet solution, you would do this using exported resources. One node could export the fact that it was an apache server and could inform nagios how to monitor it, and your nagios node(s) would import these resources and configure themselves to monitor appropriately.

However, it's not simple to remove exported resources from puppet's storeconfig. Further, it's quite often that you want to not only remove the resource from the database but also from the hosts those resources were exported to, right? We want nagios to stop monitoring that server that was decommissioned.

This demo gives an example of using an exported custom define with a timestamp attribute to signal 'expired' resources as a way of implementing such a solution.

How it works

I only use exported custom defines. That is, I never directly export a file, host, nagios_service, or package. I always wrap them in custom defines and then export that. Why? Because this lets me change the implementation of what is exported without waiting for more puppet runs and because I can add cool things like timestamps or other attributes not normally available in whatever actual resource I am using.

In this example, I wrap the 'host' resource with example::exported::expiringhost. That define (example::exported::expiringhost) wraps example::expiringhost and exports it.

This lets nodes simply do this:

example::exported::expiringhost {
  "$fqdn": ip => $ipaddress_eth0
}

And that define will export @@example::expiringhost with a timestamp of now.

About this demo

The script 'runashost.sh' runs puppet in masterless mode (applies a puppet manifest itself) and lets you spoof the hostname and the ip address of the machine to puppet so you can easily test this with multiple 'hosts' from the same machine.

Also, since I want this demo to be able to run as a normal user for testing and standalone, I choose sqlite3 to put storeconfigs in and I use the 'host' resource's “target” attribute to point at /tmp/expiring-hosts-example-output

Example run

As mentioned above, the 'hosts' file managed by this demo is: /tmp/expiring-hosts-example-output

Operations: (T is time since start in seconds)

  • (T=0) Add a host 'one.example.com'

  • (T=15) Add a host 'two.example.com'

  • (T=70) Add a host 'three.example.com'

After the 3rd host is added (at T=70), puppet will have expired 'one.example.com' for being too old and the hosts file will only contain 'two.example.com' and 'three.example.com'

% ./runashost.sh one.example.com 10.0.0.1
+ export FACTER_fqdn=one.example.com
+ export FACTER_ipaddress_eth0=10.0.0.1
+ dbflags=--dbadapter sqlite3 --dblocation storeconfigs.sqlite
+ puppet --certname one.example.com --node_name fqdn --verbose --dbadapter sqlite3 --dblocation storeconfigs.sqlite --storeconfigs --modulepath modules manifests/site.pp
info: Connecting to sqlite3 database: /home/jls/projects/puppet-examples/exported-expiration/storeconfigs.sqlite
info: Caching facts for one.example.com
notice: Scope(Example::Exported::Expiringhost[one.example.com]): Exporting host: one.example.com => 10.0.0.1
notice: Scope(Example::Expiringhost[one.example.com]): Found recently-active [one.example.com] (age: 0.2286)
info: Caching catalog for one.example.com
info: Applying configuration version '1288853587'

% ./runashost.sh two.example.com 10.0.0.2
+ export FACTER_fqdn=two.example.com
+ export FACTER_ipaddress_eth0=10.0.0.2
+ dbflags=--dbadapter sqlite3 --dblocation storeconfigs.sqlite
+ puppet --certname two.example.com --node_name fqdn --verbose --dbadapter sqlite3 --dblocation storeconfigs.sqlite --storeconfigs --modulepath modules manifests/site.pp
info: Connecting to sqlite3 database: /home/jls/projects/puppet-examples/exported-expiration/storeconfigs.sqlite
info: Caching facts for two.example.com
notice: Scope(Example::Exported::Expiringhost[two.example.com]): Exporting host: two.example.com => 10.0.0.2
notice: Scope(Example::Expiringhost[one.example.com]): Found recently-active [one.example.com] (age: 22.197815)
notice: Scope(Example::Expiringhost[two.example.com]): Found recently-active [two.example.com] (age: 0.298657)
info: Caching catalog for two.example.com
info: Applying configuration version '1288853608'
notice: /Stage[main]//Node[default]/Example::Exported::Expiringhost[two.example.com]/Example::Expiringhost[two.example.com]/Host[two.example.com]/ip: ip changed '10.0.0.1' to '10.0.0.2'
info: FileBucket adding /tmp/expiring-hosts-example-output as {md5}0cb3c66936c6ebfe7434a5979b3d6f34

% ./runashost.sh three.example.com 10.0.0.3
+ export FACTER_fqdn=three.example.com
+ export FACTER_ipaddress_eth0=10.0.0.3
+ dbflags=--dbadapter sqlite3 --dblocation storeconfigs.sqlite
+ puppet --certname three.example.com --node_name fqdn --verbose --dbadapter sqlite3 --dblocation storeconfigs.sqlite --storeconfigs --modulepath modules manifests/site.pp
info: Connecting to sqlite3 database: /home/jls/projects/puppet-examples/exported-expiration/storeconfigs.sqlite
info: Caching facts for three.example.com
notice: Scope(Example::Exported::Expiringhost[three.example.com]): Exporting host: three.example.com => 10.0.0.3
notice: Scope(Example::Expiringhost[one.example.com]): Expiring resource [one.example.com] due to age > 60 (actual: 72.42003)
notice: Scope(Example::Expiringhost[two.example.com]): Found recently-active [two.example.com] (age: 50.465772)
notice: Scope(Example::Expiringhost[three.example.com]): Found recently-active [three.example.com] (age: 0.500446)
info: Caching catalog for three.example.com
info: Applying configuration version '1288853659'
notice: /Stage[main]//Node[default]/Example::Expiringhost[one.example.com]/Host[one.example.com]/ensure: removed
info: FileBucket adding /tmp/expiring-hosts-example-output as {md5}b02db36d645e335ee0d147de653c2004
notice: /Stage[main]//Node[default]/Example::Exported::Expiringhost[three.example.com]/Example::Expiringhost[three.example.com]/Host[three.example.com]/ip: ip changed '10.0.0.2' to '10.0.0.3'

% cat /tmp/expiring-hosts-example-output
# HEADER: This file was autogenerated at Wed Nov 03 23:54:21 -0700 2010
# HEADER: by puppet.  While it can still be managed manually, it
# HEADER: is definitely not recommended.
10.0.0.2        two.example.com
10.0.0.3        three.example.com

# Now, a few minutes later, rerun 'three.example.com' and the first two hosts
# (one.example.com, two.example.com) will have expired from age.
% ./runashost.sh three.example.com 10.0.0.3
+ export FACTER_fqdn=three.example.com
+ export FACTER_ipaddress_eth0=10.0.0.3
+ dbflags=--dbadapter sqlite3 --dblocation storeconfigs.sqlite
+ puppet --certname three.example.com --node_name fqdn --verbose --dbadapter sqlite3 --dblocation storeconfigs.sqlite --storeconfigs --modulepath modules manifests/site.pp
info: Connecting to sqlite3 database: /home/jls/projects/puppet-examples/exported-expiration/storeconfigs.sqlite
notice: Scope(Example::Exported::Expiringhost[three.example.com]): Exporting host: three.example.com => 10.0.0.3
notice: Scope(Example::Expiringhost[one.example.com]): Expiring resource [one.example.com] due to age > 60 (actual: 591.973301)
notice: Scope(Example::Expiringhost[two.example.com]): Expiring resource [two.example.com] due to age > 60 (actual: 570.424178)
notice: Scope(Example::Expiringhost[three.example.com]): Found recently-active [three.example.com] (age: 1.489464)
info: Caching catalog for three.example.com
info: Applying configuration version '1288854178'
notice: /Stage[main]//Node[default]/Example::Expiringhost[two.example.com]/Host[two.example.com]/ensure: removed
info: FileBucket adding /tmp/expiring-hosts-example-output as {md5}a65177536c8f9d89044e0fa2d8cedfd9

# Voila!
% cat /tmp/expiring-hosts-example-output
# HEADER: This file was autogenerated at Thu Nov 04 00:03:00 -0700 2010
# HEADER: by puppet.  While it can still be managed manually, it
# HEADER: is definitely not recommended.
10.0.0.3        three.example.com

Notice how puppet sees that 'one.example.com' and 'two.example.com' have timestamps that are older and as a result we consider them expired.

You can use this to purge old/unused exported resources of any kind: file, user, package, service, nagios_service, etc.

Conclusion

Using timestamps makes for a cool demo, but it may not be practical in production. Obviously, 60 seconds is way too short, but a reasonable age like an hour, or a day, or a week, could work for your infrastructure. However, what if your puppet deployment is broken or hosts just can't check in for some reason?

To solve the above, you probably want to use something more explicit than 'timestamp' as an expiry value. You could support an attribute 'expired' to flag deletion instead of computing age by timestamp. Setting 'expired' would be an external action perhaps by a central truth source or by a human - today, though, this would require manipulating the database backing the storeconfigs as there is no interface in puppet (yet) for manipulating exported resources outside of a puppet run.