InstallGuide SystemMonitoring
Go up: ParentWiki
This section describes how Jabber can easily be extended for Linux system monitoring. Information in this section is not particular to Jabberd 2; however, I have included it in the guide because I have created some system administration tools that use Jabber, and I hope that others may find these scripts useful. Additionally, my scripts may possibly inspire readers to extend Jabber in new ways.
Jabber is ideally suited for system administration alerts because Jabber provides real time notification. Jabber alerts are less likely to be ignored than email alerts, they can be received anywhere the user is logged in, and Jabber alerts are free, unlike pagers.
Each subsection of this appendix describes a way that Jabber can be used for Linux system administration monitoring and alerting:
- Using the jabber_alert.pl Script
- Using Jabber for Job Monitoring
- Using Jabber for System Monitoring
- Using Jabber for System Debugging
- Using Jabber to Receive Time Sensitive Email
- Using Jabber with Mon and Nagios
All of the above tasks rely on scripts available on this site. Impatient readers may wish to peruse scripts in the tools folder before continuing.
The jabber_alert.pl script is a Perl script that sends a pre-formatted alert message via Jabber. It accepts a variety of arguments that are parsed as alert headers for a Jabber message; however, in its simplest usage it can be used to send a Jabber message from the command line:
./jabber_alert.pl -e user1@somedomain.com -n user2@somedomain.com -w user2passwdWhere -e [recipient JID] -n [sender JID] -w [sender password] [message body from STDIN]
The message body comes from standard input, and can contain as many lines as you wish. When you are ready to send the message, press CTRL-d. The -h option displays help information.
The jabber_alert.pl script can be downloaded from the tools folder. Once downloaded, you will need to chmod it (chmod a+x jabber_alert.pl) so that you can run it. jabber_alert.pl relies on these three Perl packages:
Net::Jabber
Time::Local
Getopt::Std perl Makefile.PL
make
make installOnce Net::Jabber is installed, you can use jabber_alert.pl to send Jabber messages from the command line; however, its real purpose here is to work as a Jabber alert message sender for various bash shell scripts. That is, once jabber_alert.pl is installed, it is easy to send a Jabber alert message from a shell script. All of the individual shell scripts in this section rely on jabber_alert.pl.
'''N''' Note To use it from the command line, the user password must be entered in the clear; therefore, you should be careful when using it this way, or you should use a jabber account that is not valuable.
Once jabber_alert.pl is installed, you can check out the system administration scripts I have written that use it, starting with the next section.
This section describes how to use the job_mon.sh script to monitor jobs, such as cron jobs. job_mon.sh is a script that runs a job. If the job fails (exits with a status other than 0), job_mon.sh sends an alert message via Jabber. The alert message contains information about the job and any information that the job sent to STDERR. Optionally, job_mon.sh can provide an alert message if the job completed successfully. Its usage is simple:
./job_mon.sh -j [job with or without args]The -j argument specifies the job or program for job_mon.sh to run. The job and any job arguments should be enclosed (as a single string) in quotation marks. An optional -s flag specifies that an alert should be sent on success. The -h option provides a help message.
The job_mon.sh script can be downloaded from the tools folder. Once downloaded, you will need to chmod it (chmod a+x job_mon.sh) in order to run it. job_mon.sh relies on jabber_alert.pl.
job_mon.sh requires setup for the JID of the alert sender and recipient, in addition to a password for the sender. Open job_mon.sh in an editor, and you will see the user configuration near the top of the file. You should edit the file to provide a recipient_jid, sender_jid and sender_pw. Note that recipient_jid may be the same as sender_jid. You may enter other optional information below.
'''N''' Note To use job_mon.sh, the script file must contain a Jabber user password. See the warning in the help message.
Once you have edited your Jabber information in job_mon.sh, you can easily test it. Choose a simple program to run, such as a text editor, and launch it with job_mon.sh :
./job_mon.sh -j "nedit" ps -A | grep nedit
>25430 pts/1 00:00:00 nedit
kill 25430 [13:03:38] <monitor_user@somedomain.com>
Alert: Alert Message
Time: Wed, 28 Jul 13:08:48 UTC: -4.00
User: me
Host: machine.somedomain.com
Service: nedit
Status: Job Failed: nedit terminated
with signal 15 (SIGTERM)
------------Beg-Summary------------
------------End-Summary------------
Subject: Monitor AlertI use job_mon.sh to monitor my backup jobs. For example, I run a weekly backup to tape. My fcron entry uses job_mon.sh :
0 2 * * 6 /home/scripts/job_alert.sh -s -j "/home/scripts/tape_backup_full.sh" [03:15:39] <monitor_user@somedomain.com>
Alert: Alert Message
Time: Sun, 25 Jul 03:15:38 UTC: -4.00
User: me
Host: myhost.mydomain.com
Service: /home/scripts/tape_backup_full.sh
Status: Job Completed Successfully
------------Beg-Summary------------
Rewinding tape
Tape rewind succeeded.
Mounting /boot
Boot mount succeeded.
Starting backups to tape...
Backing up: /home/
star: '/home/./me/.kde3.1/kdeinit-:0' unsupported file type 'socket'. Not dumped.
star: '/home/./me/.SB-sock' unsupported file type 'socket'. Not dumped.
star: 47590 blocks + 0 bytes (total of 1559429120 bytes = 1522880.00k).
star: The following problems occurred during archive processing:
star: Cannot: stat 0, open 0, read/write 0. Size changed 0.
star: Missing links 0, Name too long 0, File too big 0, Not dumped 2.
star: Processed all possible files, despite earlier errors.
Backups complete. Unmounting /boot
Boot unmount succeeded.
Backups complete!
------------End-Summary------------
Subject: Monitor Alert echo "Backups complete!" >&2This section describes how to use the sys_mon.sh script to receive real time Jabber alerts when a system resource is above a preset limit. When run, sys_mon.sh checks a specific system resource and checks whether that resource is above the preset threshold. Usage is based on which resource is to be checked:
sys_mon.sh -u|d|m [limit param(s)]The -d argument specifies a check on directories, where the '[limit params]' consist of threshold of kilobytes of disk space used, followed by directory(ies) under which to check, and optionally, directory(ies) to skip. Thus, when run with the -d argument, sys_mon.sh will alert for directories using more than the the threshold kilobytes.
The -m argument specifies a check for modified files, where the '[limit params]' consist of a number of minutes, followed by directory(ies) under which to check, and optionally, directory(ies) to skip. Thus, when run with the -m argument, sys_mon.sh will alert for files modified during the last (threshold) minutes under the stated directory(ies).
See the help (./sys_mon.sh -h) for more detailed usage information.
The sys_mon.sh script can be downloaded from the tools folder. Once downloaded, you will need to chmod it (chmod a+x sys_mon.sh) in order to run it. sys_mon.sh relies on jabber_alert.pl.
sys_mon.sh requires setup for the JID of the alert sender and recipient, in addition to a password for the sender. Open sys_mon.sh in an editor, and you will see the user configuration near the top of the file. You should edit the file to provide a recipient_jid, sender_jid and sender_pw. You may enter other optional information below.
'''N''' Note sys_mon.sh must contain a Jabber user password. See the warning in the help message.
Once sys_mon.sh has been setup, it can easily be tested. Run sys_mon.sh to check for file systems over 10 percent full:
./sys_mon.sh -u 10 [13:58:04] <monitor_user@somedomain.com>
Alert: Alert Message
Time: Mon, 2 Aug 13:57:59 UTC: -4.00
User: me
Host: machine.somedomain.com
Service: File System Usage Warning
------------Beg-Summary------------
File system(s) over 10%:
/ 56%
/home 79%
/home/media 41%
/mnt/temp 87%
------------End-Summary------------
Subject: Monitor Alert I use sys_mon.sh to monitor for modified files in /etc on my server. To do this, I run sys_mon.sh as a daily cron job, and set it to check for files modified during the past 1,440 minutes (1 day):
0 5 * * * /home/scripts/sys_mon.sh -m 1440 /etc -/etc/tiny/log -/etc/tiny/superviseI also use sys_mon.sh as a cron job on my workstation to catch file systems before they are completely full:
0 4 * * * /home/scripts/sys_mon.sh -u 90 [04:00:07] <monitor_user@somedomain.com>
Alert: Alert Message
Time: Mon, 2 Aug 04:00:01 UTC: -4.00
User: me
Host: machine.somedomain.com
Service: File System Usage Warning
------------Beg-Summary------------
File system(s) over 90%:
/home 94%
------------End-Summary------------
Subject: Monitor Alert Run ./sys_mon.sh -h for more examples.
This section describes how to use the sys_debug.sh script as a system diagnostic and troubleshooting tool. The sys_debug.sh script is designed to watch a process (or processes) and send a Jabber alert for log events or alert for consumption of high system resources.
sys_debug.sh differs from sys_mon.sh (in the previous section) in that sys_debug.sh is designed to run continuously as a troubleshooting tool. It monitors a process or resources that a process is using. Thus, sys_debug.sh would be useful for watching processes that are running abnormally, or for monitoring new daemons that are being put into production — as a way to ensure that the new daemons do not consume abnormal resources. Usage is based on the type of monitoring to be performed:
Usage: ./sys_debug.sh -a|-f|-p [test_param(s)]
Usage Options: ./sys_debug.sh [-s cycle_seconds] [-u exit_seconds]
[-q] [-i alert_interval] -a|-f|-p test_params The -f argument specifies monitoring of a log file, where '[test_param(s)]' are the full path the log file and a search string. sys_debug.sh will send an alert if the search string is written to the log file. On alert, the last 5 lines of the log file will be reported; however, the number of lines to report can be modified with the optional -n argument. tail and grep can be used to watch log output; what this script does is give you a real time remote alert when a specified string is written to a log. This would be useful for watching for intermittent errors.
The -p argument specifies monitoring of a process via ps , where '[test_param(s)]' consist of a choice (--pid | --ppid | -C | --User) of process filtering followed by a ps header name and threshold. sys_debug.sh can monitor any ps output that returns an integer, and it will alert when ps reports that a process — as filtered by PID, PPID, Command Name, or User — rises above the stated threshold. Multiple ps headers and thresholds are acceptable.
See the help (sys_debug.sh -h) for more detailed usage information, including information about optional arguments. See the ps man page for more information about ps headers (STANDARD FORMAT SPECIFIERS).
The sys_debug.sh script can be downloaded from the tools folder. Once downloaded, you will need to chmod it (chmod a+x sys_debug.sh) in order to run it. sys_debug.sh relies on jabber_alert.pl.
sys_debug.sh requires setup for the JID of the alert sender and recipient, in addition to a password for the sender. Open sys_debug.sh in an editor, and you will see the user configuration near the top of the file. You should edit the file to provide a recipient_jid, sender_jid and sender_pw. You may enter other optional information below.
When using sys_debug.sh, you may wish to used a fixed-width font, such as courier, in your Jabber client so that output columns align properly.
'''N''' Note sys_debug.sh must contain a Jabber user password. See the warning in the help message.
sys_debug.sh can be tested against a process running on the workstation on which it is installed by testing it against low thresholds. In this example, sys_debug.sh is run to check if any processes running for the user me are consuming more than 4% memory, 3% CPU or have caused more than 10,000 page faults:
./sys_debug.sh -p --User me %mem 4 %cpu 3 majflt 10000 Subject: Monitor Alert
[16:56:16] <monitor_user@somedomain.com>
Alert: Alert Message
Time: Mon, 2 Aug 16:56:14 UTC: -4.00
User: me
Host: machine.somedomain.com
Service: Process Alert from ./sys_debug.sh
------------Beg-Summary------------
Thresholds for majflt, %cpu, %mem reached:
( %mem over 4 )
PID CMD %mem
11227 vmware-vmx 18.4
11229 vmware-vmx 18.4
11230 vmware-vmx 18.4
11231 vmware-vmx 18.4
11232 vmware-vmx 18.4
11233 vmware-vmx 18.4
11234 vmware-vmx 18.4
11236 vmware-vmx 18.4
11237 vmware-vmx 18.4
( %cpu over 3 )
PID CMD %cpu
11237 vmware-vmx 19.0
( majflt over 10000 )
PID CMD majflt
4422 enlightenment 3893626
11227 vmware-vmx 25091
11237 vmware-vmx 44909
------------End-Summary------------
Subject: Monitor Alert Note that does not report combined memory usage by a parent PID. As far as I know, this is a limitation of Linux versions lower than 2.6.
sys_debug.sh can be used to monitor a new Jabberd 2 installation. After putting a Jabberd 2 server into production use, you might run sys_debug.sh to watch for high memory or CPU usage:
./sys_debug.sh -p --User jabber %mem 20 %cpu 20Run ./sys_debug.sh -h for more examples.
This section describes how to receive Jabber notification for time sensitive email. The email_alert.sh script forwards an email via Jabber when an arriving email matches a filter.
System administrators often use email to alert for issues that need quick resolution; however, email is not well suited for real time notifications. Another use for this script is the ability to receive important email's while you are away from your email client. For example, you may not wish to check your personal email while at work. This script would allow you to receive the most important messages via Jabber at work, while ignoring the rest until you arrive home.
The email_alert.sh script does not poll a POP server. Rather, it relies on a running email client to receive and sort mail into directories. email_alert.sh periodically checks specified directories for new email, which is then checked against a filter string. If the string is matched, the email message is forwarded via Jabber. This script is designed only to work with email messages that are stored as individual files.
email_alert.sh is designed to be run continuously in the background:
./email_alert.sh -s [seconds] &The email_alert.sh script can be downloaded from the tools folder. Once downloaded, you will need to chmod it (chmod a+x email_alert.sh) in order to run it. email_alert.sh relies on jabber_alert.pl.
email_alert.sh requires setup for the JID of the alert sender and recipient, in addition to a password for the sender. Open email_alert.sh in an editor, and you will see the Jabber configuration near the top of the file. You should edit the file to provide a recipient_jid, sender_jid and sender_pw. You may enter other optional information immediately below.
'''N''' Note email_alert.sh must contain a Jabber user password. See the warning in the help message.
Below the Jabber configuration is a section for email filters. Each filter consists of a directory and a regular expression. The intention is that you use your email client, MUA, MTA, etc., to filter your email into different directories. email_alert.sh can then scan one (or more) of these directories and look for new messages that match a corresponding regex.
For example, if you filter important email into a directory called "sysadmin", and you want instant notification for any email that contains the word "Important" in the subject line, you might set up a filter like this in email_alert.sh :
/home/me/Mail/sysadmin
^Subject: .*ImportantTo test email_alert.sh, you can set it up with a broad filter, such as the one below:
/home/me/Mail/inbox
^To:The email_alert.sh script contains some filter examples. See the section labeled, "BEGIN FILTER REGEX EXAMPLES."
This section provides some tips on how Jabber can be used with system-monitoring tools, such as Mon and Nagios. Jabber alerting is a useful feature for these tools because email alerts do not provide instant notification. Jabber provides a free instant notification anywhere the user is logged into Jabber. This is especially valuable for open source developers who, most likely, are not going to pay for a pager to receive server alerts.
This guide is intended to demonstrate how Jabber can be used with these monitoring tools. See the respective Mon and Nagios sites for instructions on how to use these applications.
Mon is a basic service monitoring tool. On startup, Mon reads a configuration file and then performs tests as scheduled in the configuration file. Mon tests consist (primarily) of Perl scripts located in /etc/mon/mon.d. If a test fails, Mon calls the alert as specified in the configuration file. Mon alerts are Perl scripts located in /etc/mon/alert.d. As of this writing, the Mon distribution comes with neither a Jabberd monitor nor a Jabber alerter; however, the Jabberdoc tools folder contains scripts for both these tasks.
The jabber_alert.pl script was originally written as an alerter for Mon. To use it with mon, simply copy it to the /etc/mon/alert.d directory, and chmod it so that it can be run. You may wish to rename it jabber.alert to maintain consistency with the other types of alerts.
Once jabber.alert is in your alert.d directory, you can use it to send Mon alerts. For example, this sample section of a mon.cf script would jab user1@somedomain.org if the web server for www.somedomain.com failed:
hostgroup webservers www.somedomain.com
watch webservers
service HTTP_Server_Check
interval 1m
monitor http.monitor -p 80 -t 30
period wd {Mon-Sun} hr {0am-24pm}
alert jabber.alert -e user1@somedomain.org -n user2@somedomain.org -w mypasswordThe tools folder also contains jabber.monitor, which is a script for monitoring Jabber servers. To use this monitor, copy it to the /etc/mon/mon.d directory and chmod it so that it can be run. Its use is similar to that of other Mon monitors. This section of a mon.cf script would test the jabber.somedomain.com server every minute:
hostgroup jabberservers jabber.somedomain.com
watch jabberservers
service Jabberd_Server_Check
interval 1m
monitor jabber.monitor -p 5222
period wd {Mon-Sun} hr {0am-24pm}
alert jabber.alert -e user@somedomain.org -n user2.somedomain.org -w mypasswordOf course, if you are monitoring a Jabber server, you want neither the Jabber alert sender nor recipient to be using the server being monitored. If you use the same Jabber server for monitoring and alerting, you would receive no alert messages if the server failed. You would want to choose a reliable Jabber server for sending and receiving alerts, and you may wish to peruse the Jabber Monitor Service, which provides graphical uptime statistics for public Jabber servers.
Mon installation and configuration is beyond the scope of this guide; however, I have included a mon.cf.sample configuration file in the tools folder. Note that this configuration relies on the alert and monitor scripts above in addition to a modified dns.monitor script. The modified dns.monitor script allows for testing of Jabber DNS SRV records.
Nagios offers many more features than Mon, including a web interface that is capable of providing graphical network maps showing problem areas. I am not yet familiar with Nagios; however, there is a Jabber alert script available for it.
© 2003 Will Kamishlian and Robert Norris
Image(http://jabberd.jabberstudio.org/2/docs/ccommons.gif, right)This work is licensed under the Creative Commons Attribution-!NonCommercial-!ShareAlike License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-sa/1.0/ or send a letter to Creative Commons, 559 Nathan Abbott Way, Stanford, California 94305, USA.
