Merged with master

README.md

@@ -49,6 +49,7 @@ Currently, we have support built in for the following alert types:
 - AWS SNS
 - VictorOps
 - PagerDuty
+- Twilio
 - Gitter
 
 Additional rule types and alerts can be easily imported or written.

docs/source/elastalert.rst

@@ -40,6 +40,7 @@ Currently, we have support built in for these alert types:
 - Slack
 - Telegram
 - Debug
+- Stomp
 
 Additional rule types and alerts can be easily imported or written. (See :ref:`Writing rule types <writingrules>` and :ref:`Writing alerts <writingalerts>`)
 
@@ -169,6 +170,9 @@ unless overwritten in the rule config. The default is "localhost".
 
 ``boto_profile``: Boto profile to use when signing requests to Amazon Elasticsearch Service, if you don't want to use the instance role keys.
 
+``replace_dots_in_field_names``: If ``True``, ElastAlert replaces any dots in field names with an underscore before writing documents to Elasticsearch.
+The default value is ``False``. Elasticsearch 2.0 - 2.3 does not support dots in field names.
+
 .. _runningelastalert:
 
 Running ElastAlert

docs/source/ruletypes.rst

@@ -870,10 +870,14 @@ This rule requires one additional option:
 
 ``fields``: A list of fields to monitor for new terms. ``query_key`` will be used if ``fields`` is not set. Each entry in the
 list of fields can itself be a list.  If a field entry is provided as a list, it will be interpreted as a set of fields
-that compose a composite key used for the ElasticSearch query.  ``Note: the composite fields may only refer to primitive
-types, otherwise the initial ElasticSearch query will not properly return the aggregation results, thus causing alerts
-to fire every time the ElastAlert service initially launches with the rule. A warning will be logged to the console if
-this scenario is encountered. However, future alerts will actually work as expected after the initial flurry.``
+that compose a composite key used for the ElasticSearch query.
+
+.. note::
+
+   The composite fields may only refer to primitive types, otherwise the initial ElasticSearch query will not properly return
+   the aggregation results, thus causing alerts to fire every time the ElastAlert service initially launches with the rule.
+   A warning will be logged to the console if this scenario is encountered. However, future alerts will actually work as
+   expected after the initial flurry.
 
 Optional:
 
@@ -1017,17 +1021,23 @@ in the case of an aggregated alert, as a JSON array, to the stdin of the process
 
 This alert requires one option:
 
-``command``: A list of arguments to execute or a string to execute. If in list format, the first argument is the name of the program to execute. If passing a
-string, the command will be executed through the shell. The command string or args will be formatted using Python's % string format syntax with the
-match passed the format argument. This means that a field can be accessed with ``%(field_name)s``. In an aggregated alert, these fields will come
-from the first match.
+``command``: A list of arguments to execute or a string to execute. If in list format, the first argument is the name of the program to execute. If passed a
+string, the command is executed through the shell.
+
+Strings can be formatted using the old-style format (``%``) or the new-style format (``.format()``). When the old-style format is used, fields are accessed
+using ``%(field_name)s``. When the new-style format is used, fields are accessed using ``{match[field_name]}``. New-style formatting allows accessing nested
+fields (e.g., ``{match[field_1_name][field_2_name]}``).
+
+In an aggregated alert, these fields come from the first match.
 
 Optional:
 
+``new_style_string_format``: If True, arguments are formatted using ``.format()`` rather than ``%``. The default is False.
+
 ``pipe_match_json``: If true, the match will be converted to JSON and passed to stdin of the command. Note that this will cause ElastAlert to block
 until the command exits or sends an EOF to stdout.
 
-Example usage::
+Example usage using old-style format::
 
     alert:
       - command
@@ -1038,6 +1048,12 @@ Example usage::
     Executing commmands with untrusted data can make it vulnerable to shell injection! If you use formatted data in
     your command, it is highly recommended that you use a args list format instead of a shell string.
 
+Example usage using new-style format::
+
+    alert:
+      - command
+    command: ["/bin/send_alert", "--username", "{match[username]}"]
+
 
 Email
 ~~~~~
@@ -1296,6 +1312,22 @@ If there's no open (i.e. unresolved) incident with this key, a new one will be c
 
 ``pagerduty_proxy``: By default ElastAlert will not use a network proxy to send notifications to Pagerduty. Set this option using ``hostname:port`` if you need to use a proxy.
 
+Twilio
+~~~~~~
+
+Twilio alerter will trigger an incident to a mobile phone as sms from your twilio phone number. Alert name will arrive as sms once this option is chosen.
+
+The alerter requires the following option:
+
+``twilio_accout_sid``: This is sid of your twilio account.
+
+``twilio_auth_token``: Auth token assosiated with your twilio account.
+
+``twilio_to_number``: The phone number where you would like send the notification.
+
+``twilio_from_number``: Your twilio phone number from which message will be sent.
+
+
 VictorOps
 ~~~~~~~~~
 
@@ -1369,6 +1401,21 @@ Debug
 
 The debug alerter will log the alert information using the Python logger at the info level. It is logged into a Python Logger object with the name ``elastalert`` that can be easily accessed using the ``getLogger`` command.
 
+Stomp
+~~~~~
+
+This alert type will use the STOMP protocol in order to push a message to a broker like ActiveMQ or RabbitMQ. The message body is a JSON string containing the alert details.
+The default values will work with a pristine ActiveMQ installation.
+
+Optional:
+
+``stomp_hostname``: The STOMP host to use, defaults to localhost.
+``stomp_hostport``: The STOMP port to use, defaults to 61613.
+``stomp_login``: The STOMP login to use, defaults to admin.
+``stomp_password``: The STOMP password to use, defaults to admin.
+``stomp_destination``: The STOMP destination to use, defaults to /queue/ALERT
+
+The stomp_destination field depends on the broker, the /queue/ALERT example is the nomenclature used by ActiveMQ. Each broker has its own logic.
 
 Alerter
 ~~~~~~~

elastalert/alerts.py

@@ -6,6 +6,8 @@
 import subprocess
 import sys
 import warnings
+import stomp
+
 from email.mime.text import MIMEText
 from email.utils import formatdate
 from smtplib import SMTP
@@ -21,6 +23,8 @@
 from requests.exceptions import RequestException
 from staticconf.loader import yaml_loader
 from texttable import Texttable
+from twilio import TwilioRestException
+from twilio.rest import TwilioRestClient
 from util import EAException
 from util import elastalert_logger
 from util import lookup_es_key
@@ -279,6 +283,53 @@ def get_account(self, account_file):
         self.password = account_conf['password']
 
 
+class StompAlerter(Alerter):
+    """ The stomp alerter publishes alerts via stomp to a broker. """
+    required_options = frozenset(['stomp_hostname', 'stomp_hostport', 'stomp_login', 'stomp_password'])
+
+    def alert(self, matches):
+
+        alerts = []
+
+        qk = self.rule.get('query_key', None)
+        fullmessage = {}
+        for match in matches:
+            if qk in match:
+                elastalert_logger.info(
+                    'Alert for %s, %s at %s:' % (self.rule['name'], match[qk], lookup_es_key(match, self.rule['timestamp_field'])))
+                alerts.append('1)Alert for %s, %s at %s:' % (self.rule['name'], match[qk], lookup_es_key(match, self.rule['timestamp_field'])))
+                fullmessage['match'] = match[qk]
+            else:
+                elastalert_logger.info('Alert for %s at %s:' % (self.rule['name'], lookup_es_key(match, self.rule['timestamp_field'])))
+                alerts.append(
+                    '2)Alert for %s at %s:' % (self.rule['name'], lookup_es_key(match, self.rule['timestamp_field']))
+                )
+                fullmessage['match'] = lookup_es_key(match, self.rule['timestamp_field'])
+            elastalert_logger.info(unicode(BasicMatchString(self.rule, match)))
+
+        fullmessage['alerts'] = alerts
+        fullmessage['rule'] = self.rule['name']
+        fullmessage['matching'] = unicode(BasicMatchString(self.rule, match))
+        fullmessage['alertDate'] = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
+        fullmessage['body'] = self.create_alert_body(matches)
+
+        self.stomp_hostname = self.rule.get('stomp_hostname', 'localhost')
+        self.stomp_hostport = self.rule.get('stomp_hostport', '61613')
+        self.stomp_login = self.rule.get('stomp_login', 'admin')
+        self.stomp_password = self.rule.get('stomp_password', 'admin')
+        self.stomp_destination = self.rule.get('stomp_destination', '/queue/ALERT')
+
+        conn = stomp.Connection([(self.stomp_hostname, self.stomp_hostport)])
+
+        conn.start()
+        conn.connect(self.stomp_login, self.stomp_password)
+        conn.send(self.stomp_destination, json.dumps(fullmessage))
+        conn.disconnect()
+
+    def get_info(self):
+        return {'type': 'stomp'}
+
+
 class DebugAlerter(Alerter):
     """ The debug alerter uses a Python logger (by default, alerting to terminal). """
 
@@ -535,6 +586,8 @@ def get_arbitrary_fields(self):
                     elif array_items == 'number':
                         self.jira_args[arg_name] = [int(v) for v in value]
                     # Also attempt to handle arrays of complex types that have to be passed as objects with an identifier 'key'
+                    elif array_items == 'option':
+                        self.jira_args[arg_name] = [{'value': v} for v in value]
                     else:
                         # Try setting it as an object, using 'name' as the key
                         # This may not work, as the key might actually be 'key', 'id', 'value', or something else
@@ -553,6 +606,8 @@ def get_arbitrary_fields(self):
                     # Number type
                     elif arg_type == 'number':
                         self.jira_args[arg_name] = int(value)
+                    elif arg_type == 'option':
+                        self.jira_args[arg_name] = {'value': value}
                     # Complex type
                     else:
                         self.jira_args[arg_name] = {'name': value}
@@ -691,18 +746,27 @@ class CommandAlerter(Alerter):
 
     def __init__(self, *args):
         super(CommandAlerter, self).__init__(*args)
+
         self.last_command = []
+
         self.shell = False
         if isinstance(self.rule['command'], basestring):
             self.shell = True
             if '%' in self.rule['command']:
                 logging.warning('Warning! You could be vulnerable to shell injection!')
             self.rule['command'] = [self.rule['command']]
 
+        self.new_style_string_format = False
+        if 'new_style_string_format' in self.rule and self.rule['new_style_string_format']:
+            self.new_style_string_format = True
+
     def alert(self, matches):
         # Format the command and arguments
         try:
-            command = [command_arg % matches[0] for command_arg in self.rule['command']]
+            if self.new_style_string_format:
+                command = [command_arg.format(match=matches[0]) for command_arg in self.rule['command']]
+            else:
+                command = [command_arg % matches[0] for command_arg in self.rule['command']]
             self.last_command = command
         except KeyError as e:
             raise EAException("Error formatting command: %s" % (e))
@@ -926,6 +990,34 @@ def get_info(self):
                 'pagerduty_client_name': self.pagerduty_client_name}
 
 
+class TwilioAlerter(Alerter):
+    required_options = frozenset(['twilio_accout_sid', 'twilio_auth_token', 'twilio_to_number', 'twilio_from_number'])
+
+    def __init__(self, rule):
+        super(TwilioAlerter, self).__init__(rule)
+        self.twilio_accout_sid = self.rule['twilio_accout_sid']
+        self.twilio_auth_token = self.rule['twilio_auth_token']
+        self.twilio_to_number = self.rule['twilio_to_number']
+        self.twilio_from_number = self.rule['twilio_from_number']
+
+    def alert(self, matches):
+        client = TwilioRestClient(self.twilio_accout_sid, self.twilio_auth_token)
+
+        try:
+            client.messages.create(body=self.rule['name'],
+                                   to=self.twilio_to_number,
+                                   from_=self.twilio_to_number)
+
+        except TwilioRestException as e:
+            raise EAException("Error posting to twilio: %s" % e)
+
+        elastalert_logger.info("Trigger sent to Twilio")
+
+    def get_info(self):
+        return {'type': 'twilio',
+                'twilio_client_name': self.twilio_from_number}
+
+
 class VictorOpsAlerter(Alerter):
     """ Creates a VictorOps Incident for each alert """
     required_options = frozenset(['victorops_api_key', 'victorops_routing_key', 'victorops_message_type'])