Table of Content:
LibreNMS includes a highly customizable alerting system.
The system requires a set of user-defined rules to evaluate the situation of each device, port, service or any other entity.
This document only covers the usage of it. See the DEVELOPMENT.md for code-documentation.
Rules are defined using a logical language.
The GUI provides a simple way of creating basic as well as complex Rules in a self-describing manner.
More complex rules can be written manually.
Rules must consist of at least 3 elements: An Entity, a Condition and a Value.
Rules can contain braces and Glues.
Entities are provided as %-Noted pair of Table and Field. For Example: %ports.ifOperStatus.
Conditions can be any of:
- Equals
= - Not Equals
!= - Matches
~ - Not Matches
!~ - Greater
> - Greater or Equal
>= - Smaller
< - Smaller or Equal
<=
Values can be Entities or any single-quoted data.
Glues can be either && for AND or || for OR.
Note: The difference between Equals and Matches (and it's negation) is that Equals does a strict comparison and Matches allows the usage of the placeholder @. The placeholder @ is comparable with .* in RegExp.
Arithmetics are allowed as well.
Alert when:
- Device goes down:
%devices.status != '1' - Any port changes:
%ports.ifOperStatus != 'up' - Root-directory gets too full:
%storage.storage_descr = '/' && %storage.storage_perc >= '75' - Any storage gets fuller than the 'warning':
%storage.storage_perc >= %storage_perc_warn
Templates can be assigned to a single or a group of rules.
They can contain any kind of text.
The template-parser understands if and foreach controls and replaces certain placeholders with information gathered about the alert.
Controls:
- if-else (Else can be omitted):
{if %placeholder == 'value'}Some Text{else}Other Text{/if} - foreach-loop:
{foreach %placeholder}Key: %key<br/>Value: %value{/foreach}
Placeholders:
- Hostname of the Device:
%hostname - Title for the Alert:
%title - Time Elapsed, Only available on recovery (
%state == 0):%elapsed - Alert-ID:
%id - Unique-ID:
%uid - Faults, Only available on alert (
%state != 0), must be iterated in a foreach ({foreach %faults}). Holds all available information about the Fault, accessable in the format%value.Column, for example:%value.ifDescr. Special field%value.stringhas most Identification-information (IDs, Names, Descrs) as single string, this is the equivalent of the default used. - State:
%state - Severity:
%severity - Rule:
%rule - Rule-Name:
%name - Timestamp:
%timestamp - Contacts, must be iterated in a foreach,
%keyholds email and%valueholds name:%contacts
The Default Template is a 'one-size-fit-all'. We highly recommend defining own templates for your rules to include more specific information. Templates can be matched against several rules.
Default Template:
%title\r\n
Severity: %severity\r\n
{if %state == 0}Time elapsed: %elapsed\r\n{/if}
Timestamp: %timestamp\r\n
Unique-ID: %uid\r\n
Rule: {if %name}%name{else}%rule{/if}\r\n
{if %faults}Faults:\r\n
{foreach %faults} #%key: %value.string\r\n{/foreach}{/if}
Alert sent to: {foreach %contacts}%value <%key> {/foreach}
Transports are located within $config['install_dir']/includes/alerts/transports.*.php and defined as well as configured via $config['alert']['transports']['Example'] = 'Some Options'.
Contacts will be gathered automatically and passed to the configured transports.
By default the Contacts will be only gathered when the alert triggers and will ignore future changes in contacts for the incident. If you want contacts to be re-gathered before each dispatch, please set $config['alert']['fixed-contacts'] = false; in your config.php.
The contacts will always include the SysContact defined in the Device's SNMP configuration and also every LibreNMS-User that has at least read-permissions on the entity that is to be alerted.
At the moment LibreNMS only supports Port or Device permissions.
You can exclude the SysContact by setting:
$config['alert']['syscontact'] = false;To include users that have Global-Read or Administrator permissions it is required to add these additions to the config.php respectively:
$config['alert']['globals'] = true; //Include Global-Read into alert-contacts
$config['alert']['admins'] = true; //Include Adminstrators into alert-contactsYou can configure these options within the WebUI now, please avoid setting these options within config.php
E-Mail transport is enabled with adding the following to your config.php:
$config['alert']['transports']['mail'] = true;The E-Mail transports uses the same email-configuration like the rest of LibreNMS.
As a small reminder, here is it's configuration directives including defaults:
$config['email_backend'] = 'mail'; // Mail backend. Allowed: "mail" (PHP's built-in), "sendmail", "smtp".
$config['email_from'] = NULL; // Mail from. Default: "ProjectName" <projectid@`hostname`>
$config['email_user'] = $config['project_id'];
$config['email_sendmail_path'] = '/usr/sbin/sendmail'; // The location of the sendmail program.
$config['email_smtp_host'] = 'localhost'; // Outgoing SMTP server name.
$config['email_smtp_port'] = 25; // The port to connect.
$config['email_smtp_timeout'] = 10; // SMTP connection timeout in seconds.
$config['email_smtp_secure'] = NULL; // Enable encryption. Use 'tls' or 'ssl'
$config['email_smtp_auth'] = FALSE; // Whether or not to use SMTP authentication.
$config['email_smtp_username'] = NULL; // SMTP username.
$config['email_smtp_password'] = NULL; // Password for SMTP authentication.
$config['alert']['default_only'] = false; //Only issue to default_mail
$config['alert']['default_mail'] = ''; //Default emailYou can configure these options within the WebUI now, please avoid setting these options within config.php
API transports definitions are a bit more complex than the E-Mail configuration.
The basis for configuration is $config['alert']['transports']['api'][METHOD] where METHOD can be get,post or put.
This basis has to contain an array with URLs of each API to call.
The URL can have the same placeholders as defined in the Template-Syntax.
If the METHOD is get, all placeholders will be URL-Encoded.
The API transport uses cURL to call the APIs, therefore you might need to install php5-curl or similar in order to make it work.
Note: it is highly recommended to define own Templates when you want to use the API transport. The default template might exceed URL-length for GET requests and therefore cause all sorts of errors.
Example:
$config['alert']['transports']['api']['get'][] = "https://api.thirdparti.es/issue?apikey=abcdefg&subject=%title";You can configure these options within the WebUI now, please avoid setting these options within config.php
The nagios transport will feed a FIFO at the defined location with the same format that nagios would.
This allows you to use other Alerting-Systems to work with LibreNMS, for example Flapjack.
$config['alert']['transports']['nagios'] = "/path/to/my.fifo"; //Flapjack expects it to be at '/var/cache/nagios3/event_stream.fifo'You can configure these options within the WebUI now, please avoid setting these options within config.php
The IRC transports only works together with the LibreNMS IRC-Bot.
Configuration of the LibreNMS IRC-Bot is described here.
$config['alert']['transports']['irc'] = true;You can configure these options within the WebUI now, please avoid setting these options within config.php
The Slack transport will POST the alert message to your Slack Incoming WebHook, you are able to specify multiple webhooks along with the relevant options to go with it. All options are optional, the only required value is for url, without this then no call to Slack will be made. Below is an example of how to send alerts to two channels with different customised options:
$config['alert']['transports']['slack'][] = array('url' => "https://hooks.slack.com/services/A12B34CDE/F56GH78JK/L901LmNopqrSTUVw2w3XYZAB4C", 'channel' => '#Alerting');
$config['alert']['transports']['slack'][] = array('url' => "https://hooks.slack.com/services/A12B34CDE/F56GH78JK/L901LmNopqrSTUVw2w3XYZAB4C", 'channel' => '@john', 'username' => 'LibreNMS', 'icon_emoji' => ':ghost:');You can configure these options within the WebUI now, please avoid setting these options within config.php
The HipChat transport requires the following:
room_id = HipChat Room ID
url = HipChat API URL+API Key
from = The name that will be displayed
The HipChat transport makes the following optional:
color = Any of HipChat's supported message colors
message_format = Any of HipChat's supported message formats
notify = 0 or 1
See the HipChat API Documentation for rooms/message for details on acceptable values.
You may notice that the link points at the "deprecated" v1 API. This is because the v2 API is still in beta.
Below are two examples of sending messages to a HipChat room.
$config['alert']['transports']['hipchat'][] = array("url" => "https://api.hipchat.com/v1/rooms/message?auth_token=9109jawregoaih",
"room_id" => "1234567",
"from" => "LibreNMS");
$config['alert']['transports']['hipchat'][] = array("url" => "https://api.hipchat.com/v1/rooms/message?auth_token=109jawregoaihj",
"room_id" => "7654321",
"from" => "LibreNMS",
"color" => "red",
"notify" => 1,
"message_format" => "text");Note: The default message format for HipChat messages is HTML. It is recommended that you specify the
textmessage format to prevent unexpected results, such as HipChat attempting to interpret angled brackets (<and>).
You can configure these options within the WebUI now, please avoid setting these options within config.php
Enabling PagerDuty transports is almost as easy as enabling email-transports.
All you need is to create a Service with type Generic API on your PagerDuty dashboard.
Now copy your API-Key from the newly created Service and setup the transport like:
$config['alert']['transports']['pagerduty'] = 'MYAPIKEYGOESHERE';That's it!
Note: Currently ACK notifications are not transported to PagerDuty, This is going to be fixed within the next major version (version by date of writing: 2015.05)
Enabling Pushover support is fairly easy, there are only two required parameters.
Firstly you need to create a new Application (called LibreNMS, for example) in your account on the Pushover website (https://pushover.net/apps)
Now copy your API Token/Key from the newly created Application and setup the transport in your config.php like:
$config['alert']['transports']['pushover'][] = array(
"appkey" => 'APPLICATIONAPIKEYGOESHERE',
"userkey" => 'USERKEYGOESHERE',
);To modify the Critical alert sound, add the 'sound_critical' parameter, example:
$config['alert']['transports']['pushover'][] = array(
"appkey" => 'APPLICATIONAPIKEYGOESHERE',
"userkey" => 'USERKEYGOESHERE',
"sound_critical" => 'siren',
);Enabling Boxcar support is super easy. Copy your access token from the Boxcar app or from the Boxcar.io website and setup the transport in your config.php like:
$config['alert']['transports']['boxcar'][] = array(
"access_token" => 'ACCESSTOKENGOESHERE',
);To modify the Critical alert sound, add the 'sound_critical' parameter, example:
$config['alert']['transports']['boxcar'][] = array(
"access_token" => 'ACCESSTOKENGOESHERE',
"sound_critical" => 'detonator-charge',
);Entities as described earlier are based on the table and column names within the database, if you are ensure of what the entity is you want then have a browse around inside MySQL using show tables and desc <tablename>.
devices.hostname = The devices hostname.
devices.location = The devices location.
devices.status = The status of the device, 1 = up, 0 = down.
devices.status_reason = The reason the device was detected as down (icmp or snmp).
devices.ignore = If the device is ignored this will be set to 1.
devices.disabled = If the device is disabled this will be set to 1.
devices.last_polled = The the last polled datetime (yyyy-mm-dd hh:mm:ss).
devices.type = The device type such as network, server, firewall, etc.
bgpPeers.astext = This is the description of the BGP Peer.
bgpPeers.bgpPeerIdentifier = The IP address of the BGP Peer.
bgpPeers.bgpPeerRemoteAs = The AS number of the BGP Peer.
bgpPeers.bgpPeerState = The operational state of the BGP session.
bgpPeers.bgpPeerAdminStatus = The administrative state of the BGP session.
bgpPeers.bgpLocalAddr = The local address of the BGP session.
ipsec_tunnels.peer_addr = The remote VPN peer address.
ipsec_tunnels.local_addr = The local VPN address.
ipsec_tunnels.tunnel_status = The VPN tunnels operational status.
mempools.mempool_type = The memory pool type such as hrstorage, cmp and cemp.
mempools.mempool_descr = The description of the pool such as Physical memory, Virtual memory and System memory.
mempools.mempool_perc = The used percentage of the memory pool.
ports.ifDescr = The interface description.
ports.ifName = The interface name.
ports.ifSpeed = The port speed in bps.
ports.ifHighSpeed = The port speed in mbps.
ports.ifOperStatus = The operational status of the port (up or down).
ports.ifAdminStatus = The administrative status of the port (up or down).
ports.ifDuplex = Duplex setting of the port.
ports.ifMtu = The MTU setting of the port.
processors.processor_usage = The usage of the processor as a percentage.
processors.processor_descr = The description of the processor.
storage.storage_descr = The description of the storage.
storage.storage_perc = The usage of the storage as a percentage.
Macros are shorthands to either portion of rules or pure SQL enhanced with placeholders.
You can define your own macros in your config.php.
Example macro-implementation of Debian-Devices
$config['alert']['macros']['rule']['is_debian'] = '%devices.features ~ "@debian@"';And in the Rule:
... && %macros.is_debian = "1" && ...
This Example-macro is a Boolean-macro, it applies a form of filter to the set of results defined by the rule. All macros that are not unary should return Boolean.
You can only apply Equal or Not-Equal Operations on Bollean-macros where True is represented by "1" and False by "0".
Device (Boolean)
Entity: %macros.device
Description: Only select devices that aren't deleted, ignored or disabled.
Source: (%devices.disabled = "0" && %devices.ignore = "0")
Device is up (Boolean)
Entity: %macros.device_up
Description: Only select devices that are up.
Implies: %macros.device
Source: (%devices.status = "1" && %macros.device)
Device is down (Boolean)
Entity: %macros.device_down
Description: Only select devices that are down.
Implies: %macros.device
Source: (%devices.status = "0" && %macros.device)
Port (Boolean)
Entity: %macros.port
Description: Only select ports that aren't deleted, ignored or disabled.
Source: (%ports.deleted = "0" && %ports.ignore = "0" && %ports.disabled = "0")
Port is up (Boolean)
Entity: %macros.port_up
Description: Only select ports that are up and also should be up.
Implies: %macros.port
Source: (%ports.ifOperStatus = "up" && %ports.ifAdminStatus = "up" && %macros.port)
Port is down (Boolean)
Entity: %macros.port_down
Description: Only select ports that are down.
Implies: %macros.port
Source: (%ports.ifOperStatus = "down" && %ports.ifAdminStatus != "down" && %macros.port)
Port-Usage in Percent (Decimal)
Entity: %macros.port_usage_perc
Description: Return port-usage in percent.
Source: ((%ports.ifInOctets_rate*8)/%ports.ifSpeed)*100
Now (Datetime)
Entity: %macros.now
Description: Alias of MySQL's NOW()
Source: NOW()
Past N Minutes (Datetime)
Entity: %macros.past_$m
Description: Returns a MySQL Timestamp dated $ Minutes in the past. $ can only be a supported Resolution.
Example: %macros.past_5m is Last 5 Minutes.
Resolution: 5,10,15,30,60
Source: DATE_SUB(NOW(),INTERVAL $ MINUTE)
Sensors (Boolean)
Entity: %macros.sensor
Description: Only select sensors that aren't ignored.
Source: (%sensors.sensor_alert = 1)
Packet Loss (Boolean)
Entity: (%macros.packet_loss_5m)
Description: Packet loss % value for the device within the last 5 minutes.
Example: %macros.packet_loss_5m > 50
Entity: (%macros.packet_loss_15m)
Description: Packet loss % value for the device within the last 15 minutes.
Example: %macros.packet_loss_15m > 50