Join GitHub today
GitHub is home to over 20 million developers working together to host and review code, manage projects, and build software together.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
|Failed to load latest commit information.|
1. Introduction. This package was created to enhance the function of munin when collecting statistics through congested networks from worker nodes distributed across North America. It specifically adds the following features: o Persistent storage for the statistics generated by the munin plugins. The plugins configured on a worker node are executed by a new script called "munin-node-redis" which saves all the data gathered in a Redis key/value (database) store. Subsequently, the data can be retrieved and saved by the munin master node when network bandwidth is available. o Because the data is no longer gatherd in real time by the munin master, a UTC timestamp is generated on the munin worker node and associated with each Redis database entry. o When creating the rrd files, this package changes the number of slots for the five minute averages. Normally, five minute averages are retained for two weeks. This package will cause five minute averges to be retained for one year. 2. Prerequisites. This package reguires the following additional packages to be in stalled prior to its installation. On the munin master node: o redis o munin-1.4.5 (munin-common-1.4.5) On a munin worker node: o redis o munin-node-1.4.5 (munin-common-1.4.5) This installation scripts of this package also assume that it will be installed on an RPM based linux distribution. 3. Prerequisite Configuration. 3.1 Munin The munin master configuration file is located at /etc/munin/munin.conf. Among other things (see munin documentation for details), this file defines the munin nodes from which data is to be collected. Normally, data is retieved from munin node server tasks running on the hosts identified. If the data is be retrieved from redis, a port sepecification of "6379" must be made for each corresponding munin node. 3.2 Munin-node The munin node configuration file is located at /etc/munin/munin-node.conf. Two parameters within this file are also used by the redis data collection task "/usr/sbin/munin-node-redis": o host_name - The advertised munin node hostname. o allow - One or more allow statements specify which munin masters may collect data from this - munin node. "/usr/sbin/munin-node-redis" will created a dedicated stack for each munin master listed in this file unless it is overridden by the /etc/munin/munin-node-redis.conf (see below). 3.3 Redis The redis configuration file is located at /etc/redis.conf. Two parameters within this file are also used by the redis data collection task "/usr/sbin/munin-node-redis": o bind - If specified, must provide the IP address accessable by each and every munin master collecting data from this munin node. o port - Must default to or specified as 6379. 4. Files. The package contains the following: ./bin/mg "mg" stands for "munin graph". This can create munin graphs for any time range that's in the rrd database. Running the command without arguments provides command information. ./bin/munin-node-redis Worker node script to run the munin plugins and gather statistics. ./cron/munin-node-redis Worker node crontab to be place in /etc/cron.d /etc/munin-node-redis.conf Sample overide file (see 4.1 below). ./INSTALL-both Installation script for a node that will be both a master and a worker node. ./INSTALL-munin Installation script for a master node. ./INSTALL-munin-node Installation script for a worker node. ./munin/1.4.5/master/lib/Munin/Master/Node.pm Master node perl library module updated to provide communication with the redis database. ./munin/1.4.5/master/lib/Munin/Master/UpdateWorker.pm Master node perl library module updated to provide communication with the redis database. ./munin/plugins/cloud_scheduler_vms_uvic UVic plugin to graph Virtual Machines on configured clouds. See "Plugin Installation" below. ./munin/plugins/condor_jobs_uvic UVic plugin to graph Condor jobs. See "Plugin Installation" below. ./munin/plugins/condor_nodes_uvic UVic plugin to graph Condor worker nodes (VMs). See "Plugin Installation" below. ./munin/plugins/if_uvic_ UVic modification to the standard "if_" plugin to change transmit and receive plotting from plus ("+")/minus ("-") to LINE/AREA plotting. See "Plugin Installation" below. ./munin/plugins/xrdfiles_uvic UVic plugin to graph XrrotD connections. Plugin uses the "lsof" command and must be run as root. See "Plugin Installation" below. ./README This README file. 5. Installation. 5.1 Worker Node. o Ensure prerequisites installed. o As root, run the INSTALL-munin-node. o Modify worker node's firewall to allow a munin master access to redis through port 6379. o If required, modify the sample /etc/munin/munin-node-redis.conf configuration file. Use this file to list the IPs of master nodes that will be retrieving data from your redis server. Normally, munin-node-redis uses the "ALLOW" statements in the standard munin-node.conf file to determine which master nodes will be retrieving data. It is assumed that all master nodes will be retrieving data through redis, and a dedicated stack is provided for each one of them. If this is not the case, this file should be used as a replacement specification so as to avoid stacking data unnecessarily. 5.2 Master Node. o Ensure prerequisites installed. o As root, run the INSTALL-munin. o Modify the master node's munin configuration (/etc/munin/munin.conf) to access redis on a munin worker. The following example illustrates a munin master that is gathering data in the traditional way from one worker node and from redis on a second worker node: # a simple host tree #[localhost] # address 127.0.0.1 # use_node_name yes [first.sample.host.ca] address 192.168.2.45 [second.sample.host.ca] address 192.168.2.46 port 6379 5.3 Plugin Installation. No installation script is provided for the plugins provided with this package. The recommended method of plugin installation is to move the required plugin to the munin-node installation directory (/usr/share/munin/plugins) and create an appropriate symbolic link from the plugin configuration directory (/etc/munin/plugins). The following links for the UVic plugins are recommended: o ln -s /usr/share/munin/plugins/cloud_scheduler_vms_uvic csvms o ln -s /usr/share/munin/plugins/condor_jobs_uvic jobs o ln -s /usr/share/munin/plugins/condor_nodes_uvic nodes o ln -s /usr/share/munin/plugins/if_uvic_ if_eth0 o ln -s /usr/share/munin/plugins/if_uvic_ if_eth1 o ln -s /usr/share/munin/plugins/if_uvic_ if_bond0 o ln -s /usr/share/munin/plugins/xrdfiles_uvic xrdfiles The xrdfiles_uvic plugin requires the "lsof" command and must be run as root by setting the "user" parameter in the plugin configuration file (/etc/munin/plugin-conf.d/munin-node. See munin documentation for further information). 6. Is it working? Use the "redis-cli" command to verify the operation of munin-node-redis and check the accessability of a munin node from a munin master. The following examples provide appropriate sample results: o redis-cli -h some.munin.node keys '*' 1. "+Plugin:cpu" 2. "+Plugin:if_eth1" 3. "hostname" 4. "+Plugin:if_eth2" 5. "+Plugin:if_bond0" 6. "22.214.171.124" 7. "+Plugin:if_eth3" o redis-cli -h some.munin.node llen 126.96.36.199 (integer) 46 o redis-cli -h some.munin.node lrange 188.8.131.52 0 4 1. if_eth2 1304978522\down.value 1336783933879\up.value 9785629692524\ 2. cpu 1304978522\user.value 21708522\nice.value 130017151\system.value 31522003\idle.value 10466661148\iowait.value 15358363\irq.value 517932\softirq.value 15426340\steal.value 0\ df 1304978522\_dev_mapper_VolGroup00_LogVol00.value 16.5813705538579\_dev_mapper_VolGroup00_LogVol02.value 0.323430150848438\_dev_sdc1.value 13.0868807827511\_dev_shm.value 0\_dev_sda1.value 36.557558432028\_dev_sdb1.value 23.6127534433507\ 3. if_eth3 1304978522\down.value 1085119065088\up.value 8893652594066\ 4. if_bond0 1304978522\down.value 4752135687971\up.value 56663271471990\ 5. if_eth1 1304978522\down.value 1792681287232\up.value 13181952356502\ 7. Checkiing and resetting plugin configuration information. Munin masters required plugins to provide configuration information when called with the "config" parameter, and data values otherwise (see munin documentation for more information). The muni-node-redis package provides plugin configuration information to munin masters via static variables within the redis database. An attempt is made to verify the saved configuration each time data values are stored. However, it is possible that plugin configuration changes are not detected that the static variables become stale. A single variable is created for each plugin (see "Is it working?" above). To review the saved configuration information for any specific plugin, use the following command: o redis-cli -h some.munin.node get "+Plugin:cpu" "graph_title CPU usage\\graph_order system user nice idle iowait irq softirq\\graph_args --base 1000 -r --lower-limit 0 --upper-limit 1600\\graph_vlabel %\\graph_scale no\\graph_info This graph shows how CPU time is spent.\\graph_category system\\graph_period second\\system.label system\\system.draw AREA\\system.min 0\\system.type DERIVE\\system.info CPU time spent by the kernel in system activities\\user.label user\\user.draw STACK\\user.min 0\\user.type DERIVE\\user.info CPU time spent by normal programs and daemons\\nice.label nice\\nice.draw STACK\\nice.min 0\\nice.type DERIVE\\nice.info CPU time spent by nice(1)d programs\\idle.label idle\\idle.draw STACK\\idle.min 0\\idle.type DERIVE\\idle.info Idle CPU time\\iowait.label iowait\\iowait.draw STACK\\iowait.min 0\\iowait.type DERIVE\\iowait.info CPU time spent waiting for I/O operations to finish when there is nothing else to do.\\irq.label irq\\irq.draw STACK\\irq.min 0\\irq.type DERIVE\\irq.info CPU time spent handling interrupts\\softirq.label softirq\\softirq.draw STACK\\softirq.min 0\\softirq.type DERIVE\\softirq.info CPU time spent handling \"batched\" interrupts\\steal.label steal\\steal.draw STACK\\steal.min 0\\steal.type DERIVE\\steal.info The time that a virtual CPU had runnable tasks, but the virtual CPU itself was not running\\ To force a reset of all plugin configuration variables, issue the command: o redis-cli -h some.munin.node del hostname