Munin patches to allow caching and persistence via Redis DB
Perl Shell
Switch branches/tags
Nothing to show
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
bin
cron
etc
munin
INSTALL-both
INSTALL-munin
INSTALL-munin-node
README

README

1. Introduction.

This package was created to enhance the function of munin when collecting statistics through
congested networks from worker nodes distributed across North America. It specifically adds
the following features:

o Persistent storage for the statistics generated by the munin plugins. The plugins configured
  on a worker node are executed by a new script called "munin-node-redis" which saves all the
  data gathered in a Redis key/value (database) store. Subsequently, the data can be retrieved
  and saved by the munin master node when network bandwidth is available.

o Because the data is no longer gatherd in real time by the munin master, a UTC timestamp is
  generated on the munin worker node and associated with each Redis database entry.

o When creating the rrd files, this package changes the number of slots for the five minute
  averages. Normally, five minute averages are retained for two weeks. This package will 
  cause five minute averges to be retained for one year.

2. Prerequisites.

This package reguires the following additional packages to be in stalled prior to its
installation. 

On the munin master node:

o redis
o munin-1.4.5 (munin-common-1.4.5)

On a munin worker node:

o redis
o munin-node-1.4.5 (munin-common-1.4.5)

This installation scripts of this package also assume that it will be installed on an RPM
based linux distribution. 

3. Prerequisite Configuration.

3.1 Munin

The munin master configuration file is located at /etc/munin/munin.conf. Among other things (see
munin documentation for details), this file defines the munin nodes from which data is to be collected.
Normally, data is retieved from munin node server tasks running on the hosts identified. If the data is 
be retrieved from redis, a port sepecification of "6379" must be made for each corresponding munin node. 

3.2 Munin-node

The munin node configuration file is located at /etc/munin/munin-node.conf. Two parameters within this
file are also used by the redis data collection task "/usr/sbin/munin-node-redis":

o host_name    - The advertised munin node hostname.
o allow        - One or more allow statements specify which munin masters may collect data from this
               - munin node. "/usr/sbin/munin-node-redis" will created a dedicated stack for each munin
			     master listed in this file unless it is overridden by the /etc/munin/munin-node-redis.conf
				 (see below).

3.3 Redis

The redis configuration file is located at /etc/redis.conf. Two parameters within this file are also
used by the redis data collection task "/usr/sbin/munin-node-redis":

o bind         - If specified, must provide the IP address accessable by each and every munin
                 master collecting data from this munin node.
o port         - Must default to or specified as 6379.


4. Files.

The package contains the following:

./bin/mg												"mg" stands for "munin graph". This can create
														munin graphs for any time range that's in the rrd
														database. Running the command without arguments
														provides command information.

./bin/munin-node-redis									Worker node script to run the munin plugins and 
															gather statistics.

./cron/munin-node-redis									Worker node crontab to be place in /etc/cron.d

/etc/munin-node-redis.conf								Sample overide file (see 4.1 below).

./INSTALL-both											Installation script for a node that will be both
															a master and a worker node.

./INSTALL-munin											Installation script for a master node.

./INSTALL-munin-node									Installation script for a worker node.

./munin/1.4.5/master/lib/Munin/Master/Node.pm			Master node perl library module updated to provide
															communication with the redis database.

./munin/1.4.5/master/lib/Munin/Master/UpdateWorker.pm	Master node perl library module updated to provide
															communication with the redis database.

./munin/plugins/cloud_scheduler_vms_uvic				UVic plugin to graph Virtual Machines on configured
															clouds. See "Plugin Installation" below.

./munin/plugins/condor_jobs_uvic						UVic plugin to graph Condor jobs. See "Plugin
															Installation" below.

./munin/plugins/condor_nodes_uvic						UVic plugin to graph Condor worker nodes (VMs). See
                                                            "Plugin Installation" below.

./munin/plugins/if_uvic_								UVic modification to the standard "if_" plugin to 
															change transmit and receive plotting from
															plus ("+")/minus ("-") to LINE/AREA plotting.
                                                            See "Plugin Installation" below.

./munin/plugins/xrdfiles_uvic							UVic plugin to graph XrrotD connections. Plugin 
															uses the "lsof" command and must be run as
															root. See "Plugin Installation" below.

./README												This README file.

5. Installation.

5.1 Worker Node.

o Ensure prerequisites installed.
o As root, run the INSTALL-munin-node.
o Modify worker node's firewall to allow a munin master access to redis through port 6379.
o If required, modify the sample /etc/munin/munin-node-redis.conf configuration file. Use this file
  to list the IPs of master nodes that will be retrieving data from your redis server. Normally,
  munin-node-redis uses the "ALLOW" statements in the standard munin-node.conf file to determine
  which master nodes will be retrieving data. It is assumed that all master nodes will be retrieving
  data through redis, and a dedicated stack is provided for each one of them. If this is not the
  case, this file should be used as a replacement specification so as to avoid stacking data
  unnecessarily.

5.2 Master Node.

o Ensure prerequisites installed.
o As root, run the INSTALL-munin.
o Modify the master node's munin configuration (/etc/munin/munin.conf) to access redis
on a munin worker. The following example illustrates a munin master that is gathering data
in the traditional way from one worker node and from redis on a second worker node:

# a simple host tree
#[localhost]
#   address 127.0.0.1
#   use_node_name yes

[first.sample.host.ca]
   address 192.168.2.45

[second.sample.host.ca]
   address 192.168.2.46
      port 6379

5.3 Plugin Installation.

No installation script is provided for the plugins provided with this package. The recommended method of plugin 
installation is to move the required plugin to the munin-node installation directory (/usr/share/munin/plugins) 
and create an appropriate symbolic link from the plugin configuration directory (/etc/munin/plugins). The
following links for the UVic plugins are recommended:

o ln -s /usr/share/munin/plugins/cloud_scheduler_vms_uvic csvms
o ln -s /usr/share/munin/plugins/condor_jobs_uvic jobs
o ln -s /usr/share/munin/plugins/condor_nodes_uvic nodes
o ln -s /usr/share/munin/plugins/if_uvic_ if_eth0
o ln -s /usr/share/munin/plugins/if_uvic_ if_eth1
o ln -s /usr/share/munin/plugins/if_uvic_ if_bond0
o ln -s /usr/share/munin/plugins/xrdfiles_uvic xrdfiles

The xrdfiles_uvic plugin requires the "lsof" command and must be run as root by setting the "user" 
parameter in the plugin configuration file (/etc/munin/plugin-conf.d/munin-node. See munin documentation
for further information).

6. Is it working?

Use the "redis-cli" command to verify the operation of munin-node-redis and check the accessability of
a munin node from a munin master. The following examples provide appropriate sample results:

o redis-cli -h some.munin.node keys '*'
	1. "+Plugin:cpu"
	2. "+Plugin:if_eth1"
	3. "hostname"
	4. "+Plugin:if_eth2"
	5. "+Plugin:if_bond0"
	6. "123.12.124.1"
	7. "+Plugin:if_eth3"

o redis-cli -h some.munin.node llen 123.12.124.1
	(integer) 46

o redis-cli -h some.munin.node lrange 123.12.124.1 0 4
	1. if_eth2 1304978522\down.value 1336783933879\up.value 9785629692524\
	2. cpu 1304978522\user.value 21708522\nice.value 130017151\system.value 31522003\idle.value 10466661148\iowait.value 15358363\irq.value 517932\softirq.value 15426340\steal.value 0\
	df 1304978522\_dev_mapper_VolGroup00_LogVol00.value 16.5813705538579\_dev_mapper_VolGroup00_LogVol02.value 0.323430150848438\_dev_sdc1.value 13.0868807827511\_dev_shm.value 0\_dev_sda1.value 36.557558432028\_dev_sdb1.value 23.6127534433507\
	3. if_eth3 1304978522\down.value 1085119065088\up.value 8893652594066\
	4. if_bond0 1304978522\down.value 4752135687971\up.value 56663271471990\
	5. if_eth1 1304978522\down.value 1792681287232\up.value 13181952356502\

7. Checkiing and resetting plugin configuration information.

Munin masters required plugins to provide configuration information when called with the "config" parameter,
and data values otherwise (see munin documentation for more information). The muni-node-redis package provides 
plugin configuration information to munin masters via static variables within the redis database. An attempt
is made to verify the saved configuration each time data values are stored. However, it is possible that
plugin configuration changes are not detected that the static variables become stale. A single variable is
created for each plugin (see "Is it working?" above). To review the saved configuration information for any
specific plugin, use the following command:

o redis-cli -h some.munin.node get "+Plugin:cpu"
	"graph_title CPU usage\\graph_order system user nice idle iowait irq softirq\\graph_args --base 1000 -r --lower-limit 0 --upper-limit 1600\\graph_vlabel %\\graph_scale no\\graph_info This graph shows how CPU time is spent.\\graph_category system\\graph_period second\\system.label system\\system.draw AREA\\system.min 0\\system.type DERIVE\\system.info CPU time spent by the kernel in system activities\\user.label user\\user.draw STACK\\user.min 0\\user.type DERIVE\\user.info CPU time spent by normal programs and daemons\\nice.label nice\\nice.draw STACK\\nice.min 0\\nice.type DERIVE\\nice.info CPU time spent by nice(1)d programs\\idle.label idle\\idle.draw STACK\\idle.min 0\\idle.type DERIVE\\idle.info Idle CPU time\\iowait.label iowait\\iowait.draw STACK\\iowait.min 0\\iowait.type DERIVE\\iowait.info CPU time spent waiting for I/O operations to finish when there is nothing else to do.\\irq.label irq\\irq.draw STACK\\irq.min 0\\irq.type DERIVE\\irq.info CPU time spent handling interrupts\\softirq.label softirq\\softirq.draw STACK\\softirq.min 0\\softirq.type DERIVE\\softirq.info CPU time spent handling \"batched\" interrupts\\steal.label steal\\steal.draw STACK\\steal.min 0\\steal.type DERIVE\\steal.info The time that a virtual CPU had runnable tasks, but the virtual CPU itself was not running\\

To force a reset of all plugin configuration variables, issue the command:

o redis-cli -h some.munin.node del hostname