Skip to content

Commit

Permalink
elasticsearch -> Elasticsearch
Browse files Browse the repository at this point in the history
  • Loading branch information
bean5 committed Sep 22, 2016
1 parent 8c8ffed commit d1fd90c
Show file tree
Hide file tree
Showing 25 changed files with 91 additions and 99 deletions.
2 changes: 1 addition & 1 deletion README.md
Expand Up @@ -4,7 +4,7 @@
[![Join the chat at https://gitter.im/Yelp/elastalert](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/Yelp/elastalert?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)

## ElastAlert - [Read the Docs](http://elastalert.readthedocs.org).
### Easy & Flexible Alerting With ElasticSearch
### Easy & Flexible Alerting With Elasticsearch

ElastAlert is a simple framework for alerting on anomalies, spikes, or other patterns of interest from data in Elasticsearch.

Expand Down
16 changes: 8 additions & 8 deletions config.yaml.example
Expand Up @@ -2,7 +2,7 @@
# Any .yaml file will be loaded as a rule
rules_folder: example_rules

# How often ElastAlert will query elasticsearch
# How often ElastAlert will query Elasticsearch
# The unit can be anything from weeks to seconds
run_every:
minutes: 1
Expand All @@ -12,29 +12,29 @@ run_every:
buffer_time:
minutes: 15

# The elasticsearch hostname for metadata writeback
# Note that every rule can have its own elasticsearch host
# The Elasticsearch hostname for metadata writeback
# Note that every rule can have its own Elasticsearch host
es_host: elasticsearch.example.com

# The elasticsearch port
# The Elasticsearch port
es_port: 9200

# Optional URL prefix for elasticsearch
# Optional URL prefix for Elasticsearch
#es_url_prefix: elasticsearch

# Connect with TLS to elasticsearch
# Connect with TLS to Elasticsearch
#use_ssl: True

# Verify TLS certificates
#verify_certs: True

# GET request with body is the default option for Elasticsearch.
# GET request with body is the default option for Elasticsearch.
# If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
# See http://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport
# for details
#es_send_get_body_as: GET

# Option basic-auth username and password for elasticsearch
# Option basic-auth username and password for Elasticsearch
#es_username: someusername
#es_password: somepassword

Expand Down
10 changes: 5 additions & 5 deletions docs/source/elastalert.rst
@@ -1,4 +1,4 @@
ElastAlert - Easy & Flexible Alerting With ElasticSearch
ElastAlert - Easy & Flexible Alerting With Elasticsearch
********************************************************

ElastAlert is a simple framework for alerting on anomalies, spikes, or other patterns of interest from data in Elasticsearch.
Expand Down Expand Up @@ -137,9 +137,9 @@ configuration.

``max_query_size``: The maximum number of documents that will be downloaded from Elasticsearch in a single query. The
default is 10,000, and if you expect to get near this number, consider using ``use_count_query`` for the rule. If this
limit is reached, ElastAlert will `scroll <https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-scroll.html>`_ through pages the size of ``max_query_size`` until processing all results.
limit is reached, ElastAlert will `scroll <https://www.elastic.co/guide/en/Elasticsearch/reference/current/search-request-scroll.html>`_ through pages the size of ``max_query_size`` until processing all results.

``scroll_keepalive``: The maximum time (formatted in `Time Units <https://www.elastic.co/guide/en/elasticsearch/reference/current/common-options.html#time-units>`_) the scrolling context should be kept alive. Avoid using high values as it abuses resources in ElasticSearch, but be mindful to allow sufficient time to finish processing all the results.
``scroll_keepalive``: The maximum time (formatted in `Time Units <https://www.elastic.co/guide/en/Elasticsearch/reference/current/common-options.html#time-units>`_) the scrolling context should be kept alive. Avoid using high values as it abuses resources in Elasticsearch, but be mindful to allow sufficient time to finish processing all the results.

``max_aggregation``: The maximum number of alerts to aggregate together. If a rule has ``aggregation`` set, all
alerts occuring within a timeframe will be sent together. The default is 10,000.
Expand All @@ -165,9 +165,9 @@ unless overwritten in the rule config. The default is "localhost".

``email_reply_to``: This sets the Reply-To header in emails. The default is the recipient address.

``aws_region``: This makes ElastAlert to sign HTTP requests when using Amazon ElasticSearch Service. It'll use instance role keys to sign the requests.
``aws_region``: This makes ElastAlert to sign HTTP requests when using Amazon Elasticsearch Service. It'll use instance role keys to sign the requests.

``boto_profile``: Boto profile to use when signing requests to Amazon ElasticSearch Service, if you don't want to use the instance role keys.
``boto_profile``: Boto profile to use when signing requests to Amazon Elasticsearch Service, if you don't want to use the instance role keys.

.. _runningelastalert:

Expand Down
2 changes: 1 addition & 1 deletion docs/source/elastalert_status.rst
Expand Up @@ -26,7 +26,7 @@ elastalert_status
- ``matches``: The number of matches that the rule returned after processing the hits. Note that this does not necessarily mean that alerts were triggered.
- ``time_taken``: The number of seconds it took for this query to run.

``elastalert_status`` is what ElastAlert will use to determine what time range to query when it first starts to avoid duplicating queries.
``elastalert_status`` is what ElastAlert will use to determine what time range to query when it first starts to avoid duplicating queries.
For each rule, it will start querying from the most recent endtime. If ElastAlert is running in debug mode, it will still attempt to base
its start time by looking for the most recent search performed, but it will not write the results of any query back to Elasticsearch.

Expand Down
2 changes: 1 addition & 1 deletion docs/source/index.rst
Expand Up @@ -3,7 +3,7 @@
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
ElastAlert - Easy & Flexible Alerting With ElasticSearch
ElastAlert - Easy & Flexible Alerting With Elasticsearch
========================================================

Contents:
Expand Down
17 changes: 6 additions & 11 deletions docs/source/recipes/adding_alerts.rst
Expand Up @@ -16,7 +16,7 @@ something like this:
def get_info(self):
...
You can import alert types by specifying the type as ``module.file.AlertName``, where module is the name of a python module,
You can import alert types by specifying the type as ``module.file.AlertName``, where module is the name of a python module,
and file is the name of the python file containing a ``Alerter`` subclass named ``AlertName``.

Basics
Expand All @@ -41,7 +41,7 @@ alert(self, match):
-------------------

ElastAlert will call this function to send an alert. ``matches`` is a list of dictionary objects with
information about the match. You can get a nice string representation of the match by calling
information about the match. You can get a nice string representation of the match by calling
``self.rule['type'].get_match_str(match, self.rule)``. If this method raises an exception, it will
be caught by ElastAlert and the alert will be marked as unsent and saved for later.

Expand Down Expand Up @@ -74,11 +74,11 @@ Now, in a file named ``my_alerts.py``, add
# By setting required_options to a set of strings
# You can ensure that the rule config file specifies all
# of the options. Otherwise, ElastAlert will throw an exception
# of the options. Otherwise, ElastAlert will throw an exception
# when trying to load the rule.
required_options = set(['output_file_path'])
# Alert is called
# Alert is called
def alert(self, matches):
# Matches is a list of match dictionaries.
Expand All @@ -88,11 +88,11 @@ Now, in a file named ``my_alerts.py``, add
# Config options can be accessed with self.rule
with open(self.rule['output_file_path'], "a") as output_file:
# basic_match_string will transform the match into the default
# human readable string format
match_string = str(BasicMatchString(self.rule, match))
output_file.write(match_string)
# get_info is called after an alert is sent to get data that is written back
Expand All @@ -112,8 +112,3 @@ In the rule configuration file, we are going to specify the alert by writing
ElastAlert will attempt to import the alert with ``from elastalert_modules.my_alerts import AwesomeNewAlerter``.
This means that the folder must be in a location where it can be imported as a python module.





12 changes: 5 additions & 7 deletions docs/source/recipes/adding_rules.rst
Expand Up @@ -45,7 +45,7 @@ add_data(self, data):

When ElastAlert queries Elasticsearch, it will pass all of the hits to the rule type by calling ``add_data``.
``data`` is a list of dictionary objects which contain all of the fields in ``include``, ``query_key`` and ``compare_key``
if they exist, and ``@timestamp`` as a datetime object. They will always come in chronological order sorted by '@timestamp'.
if they exist, and ``@timestamp`` as a datetime object. They will always come in chronological order sorted by '@timestamp'.

get_match_str(self, match):
------------------------------
Expand Down Expand Up @@ -91,7 +91,7 @@ Now, in a file named ``my_rules.py``, add
# By setting required_options to a set of strings
# You can ensure that the rule config file specifies all
# of the options. Otherwise, ElastAlert will throw an exception
# of the options. Otherwise, ElastAlert will throw an exception
# when trying to load the rule.
required_options = set(['time_start', 'time_end', 'usernames'])
Expand All @@ -100,7 +100,7 @@ Now, in a file named ``my_rules.py``, add
# including all the fields that the config specifies with "include"
def add_data(self, data):
for document in data:
# To access config options, use self.rules
if document['username'] in self.rules['usernames']:
Expand All @@ -110,10 +110,10 @@ Now, in a file named ``my_rules.py``, add
# Convert time_start and time_end to time objects
time_start = dateutil.parser.parse(self.rules['time_start']).time()
time_end = dateutil.parser.parse(self.rules['time_end']).time()
# If the time falls between start and end
if login_time > time_start and login_time < time_end:
# To add a match, use self.add_match
self.add_match(document)
Expand Down Expand Up @@ -162,5 +162,3 @@ An alert from this rule will look something like::

@timestamp: 2015-03-02T22:23:24Z
username: userXYZ


17 changes: 8 additions & 9 deletions docs/source/recipes/signing_requests.rst
@@ -1,22 +1,21 @@
.. _signingrequests:

Signing requests to Amazon ElasticSearch service
Signing requests to Amazon Elasticsearch service
============

When using Amazon ElasticSearch service, you need to secure your ElasticSearch from the outside.
Currently, there is no way to secure your ElasticSearch using network firewall rules, so the only way is to signing the requests using the access key and secret key for a role or user with permissions on the ElasticSearch service.
When using Amazon Elasticsearch service, you need to secure your Elasticsearch from the outside.
Currently, there is no way to secure your Elasticsearch using network firewall rules, so the only way is to signing the requests using the access key and secret key for a role or user with permissions on the Elasticsearch service.

We offer two different options to sign ElastAlert requests to ElasticSearch: using instance roles and boto profiles.
We offer two different options to sign ElastAlert requests to Elasticsearch: using instance roles and boto profiles.

Using instance role
-------------------
Typically, you'll deploy ElastAlert on a running EC2 instance on AWS. You can assign a role to this instance that gives it permissions to read from and write to the ElasticSearch service.
Then you just need to add the ``aws_region`` option to the configuration file. This will tell ElastAlert to sign the requests to ElasticSearch.
Typically, you'll deploy ElastAlert on a running EC2 instance on AWS. You can assign a role to this instance that gives it permissions to read from and write to the Elasticsearch service.
Then you just need to add the ``aws_region`` option to the configuration file. This will tell ElastAlert to sign the requests to Elasticsearch.

Using boto profiles
--------------------
You can also create a user with permissions on the ElasticSearch service and tell ElastAlert to authenticate itself using that user.
You can also create a user with permissions on the Elasticsearch service and tell ElastAlert to authenticate itself using that user.
First, create a boto profile in the machine where you'd like to run ElastAlert for the user with permissions. Then, just add two options to the configuration file:
- ``aws_region``: that tells ElastAlert to sign the requests to ElasticSearch. It's the AWS region where you want to operate.
- ``aws_region``: that tells ElastAlert to sign the requests to Elasticsearch. It's the AWS region where you want to operate.
- ``boto_profile``: with the name of the boto profile to use to sign the requests.

8 changes: 4 additions & 4 deletions docs/source/recipes/writing_filters.rst
Expand Up @@ -5,8 +5,8 @@ Writing Filters For Rules

This document describes how to create a filter section for your rule config file.

The filters used in rules are part of the Elasticsearch query DSL, further documentation for which can be found at
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl.html
The filters used in rules are part of the Elasticsearch query DSL, further documentation for which can be found at
http://www.Elasticsearch.org/guide/en/Elasticsearch/reference/current/query-dsl.html
This document contains a small subset of particularly useful filters.

The filter section is passed to Elasticsearch exactly as follows::
Expand Down Expand Up @@ -54,7 +54,7 @@ The term type allows for exact field matches::

Note that a term query may not behave as expected if a field is analyzed. By default, many string fields will be tokenized by whitespace, and a term query for "foo bar" may not match
a field that appears to have the value "foo bar", unless it is not analyzed. Conversely, a term query for "foo" will match analyzed strings "foo bar" and "foo baz". For full text
matching on analyzed fields, use query_string. See http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/term-vs-full-text.html
matching on analyzed fields, use query_string. See http://www.Elasticsearch.org/guide/en/Elasticsearch/guide/current/term-vs-full-text.html

terms
*****
Expand Down Expand Up @@ -129,7 +129,7 @@ ElastAlert will exit with an error like "Could not download filters for .."
The second way is to generate a config file once using the kibana dashboard. To do this, run ``elastalert-rule-from-kibana``.

.. code-block:: console
$ elastalert-rule-from-kibana
Elasticsearch host: elasticsearch.example.com
Elasticsearch port: 14900
Expand Down
14 changes: 7 additions & 7 deletions docs/source/ruletypes.rst
Expand Up @@ -416,7 +416,7 @@ match_enhancements

``match_enhancements``: A list of enhancement modules to use with this rule. An enhancement module is a subclass of enhancements.BaseEnhancement
that will be given the match dictionary and can modify it before it is passed to the alerter. The enhancements will be run after silence and realert
is calculated and in the case of aggregated alerts, right before the alert is sent. This can be changed by setting ``run_enhancements_first``.
is calculated and in the case of aggregated alerts, right before the alert is sent. This can be changed by setting ``run_enhancements_first``.
The enhancements should be specified as
``module.file.EnhancementName``. See :ref:`Enhancements` for more information. (Optional, list of strings, no default)

Expand Down Expand Up @@ -459,7 +459,7 @@ _source_enabled

``_source_enabled``: If true, ElastAlert will use _source to retrieve fields from documents in Elasticsearch. If false,
ElastAlert will use ``fields`` to retrieve stored fields. Both of these are represented internally as if they came from ``_source``.
See https://www.elastic.co/guide/en/elasticsearch/reference/1.3/mapping-fields.html for more details. The fields used come from ``include``,
See https://www.elastic.co/guide/en/Elasticsearch/reference/1.3/mapping-fields.html for more details. The fields used come from ``include``,
see above for more details. (Optional, boolean, default True)

Some rules and alerts require additional options, which also go in the top level of the rule configuration file.
Expand Down Expand Up @@ -643,7 +643,7 @@ This rule requires two additional options:

Optional:

``use_count_query``: If true, ElastAlert will poll elasticsearch using the count api, and not download all of the matching documents. This is
``use_count_query``: If true, ElastAlert will poll Elasticsearch using the count api, and not download all of the matching documents. This is
useful is you care only about numbers and not the actual data. It should also be used if you expect a large number of query hits, in the order
of tens of thousands or more. ``doc_type`` must be set to use this.

Expand Down Expand Up @@ -772,7 +772,7 @@ consider the following examples::
trigger an immediate alert. When set to false, baseline must be established for each new ``query_key`` value, and then subsequent spikes may
cause alerts. Baseline is established after ``timeframe`` has elapsed twice since first occurrence.

``use_count_query``: If true, ElastAlert will poll elasticsearch using the count api, and not download all of the matching documents. This is
``use_count_query``: If true, ElastAlert will poll Elasticsearch using the count api, and not download all of the matching documents. This is
useful is you care only about numbers and not the actual data. It should also be used if you expect a large number of query hits, in the order
of tens of thousands or more. ``doc_type`` must be set to use this.

Expand Down Expand Up @@ -824,9 +824,9 @@ This rule requires one additional option:

``fields``: A list of fields to monitor for new terms. ``query_key`` will be used if ``fields`` is not set. Each entry in the
list of fields can itself be a list. If a field entry is provided as a list, it will be interpreted as a set of fields
that compose a composite key used for the elasticsearch query. ``Note: the composite fields may only refer to primitive
types, otherwise the initial elasticsearch query will not properly return the aggregation results, thus causing alerts
to fire every time the elastalert service initially launches with the rule. A warning will be logged to the console if
that compose a composite key used for the Elasticsearch query. ``Note: the composite fields may only refer to primitive
types, otherwise the initial Elasticsearch query will not properly return the aggregation results, thus causing alerts
to fire every time the ElastAlert service initially launches with the rule. A warning will be logged to the console if
this scenario is encountered. However, future alerts will actually work as expected after the initial flurry.``

Optional:
Expand Down

0 comments on commit d1fd90c

Please sign in to comment.