Permalink
Browse files

Merge pull request #988 from mozilla/create_mozdef_util_docs

Create mozdef util docs
  • Loading branch information...
pwnbus committed Nov 29, 2018
2 parents ca362d6 + dd78571 commit a0dd2d79c1de609039649d12b9058782188d795f
@@ -0,0 +1,2 @@
sphinx
sphinx_rtd_theme
@@ -97,7 +97,7 @@
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
html_theme = 'default'
html_theme = 'sphinx_rtd_theme'
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
@@ -14,6 +14,7 @@ Table of Contents
demo
installation
alert_development_guide
mozdef_util
screenshots
usage
cloud_deployment
@@ -0,0 +1,9 @@
Mozdef_util Library
===================
We provide a library used to interact with MozDef components.
.. include:: mozdef_util/connect.rst
.. include:: mozdef_util/create.rst
.. include:: mozdef_util/search.rst
.. include:: mozdef_util/match_query_classes.rst
@@ -0,0 +1,8 @@
Connecting to Elasticsearch
---------------------------
.. code-block:: python
:linenos:
from mozdef_util.elasticsearch_client import ElasticsearchClient
es_client = ElasticsearchClient("http://127.0.0.1:9200")
@@ -0,0 +1,85 @@
Creating/Updating Documents
---------------------------
Create a new Event
^^^^^^^^^^^^^^^^^^
.. code-block:: python
:linenos:
event_dict = {
"example_key": "example value"
}
es_client.save_event(body=event_dict)
Update an existing event
^^^^^^^^^^^^^^^^^^^^^^^^
.. code-block:: python
:linenos:
event_dict = {
"example_key": "example new value"
}
# Assuming 12345 is the id of the existing entry
es_client.save_event(body=event_dict, doc_id="12345")
Create a new alert
^^^^^^^^^^^^^^^^^^
.. code-block:: python
:linenos:
alert_dict = {
"example_key": "example value"
}
es_client.save_alert(body=alert_dict)
Update an existing alert
^^^^^^^^^^^^^^^^^^^^^^^^
.. code-block:: python
:linenos:
alert_dict = {
"example_key": "example new value"
}
# Assuming 12345 is the id of the existing entry
es_client.save_alert(body=alert_dict, doc_id="12345")
Create a new generic document
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. code-block:: python
:linenos:
document_dict = {
"example_key": "example value"
}
es_client.save_object(index='randomindex', doc_type='randomtype', body=document_dict)
Update an existing document
^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. code-block:: python
:linenos:
document_dict = {
"example_key": "example new value"
}
# Assuming 12345 is the id of the existing entry
es_client.save_object(index='randomindex', doc_type='randomtype', body=document_dict, doc_id="12345")
Bulk Importing
^^^^^^^^^^^^^^
.. code-block:: python
:linenos:
from mozdef_util.elasticsearch_client import ElasticsearchClient
es_client = ElasticsearchClient("http://127.0.0.1:9200", bulk_amount=30, bulk_refresh_time=5)
es_client.save_event(body={'key': 'value'}, bulk=True)
- Line 2: bulk_amount (defaults to 100), specifies how many messages should sit in the bulk queue before they get written to elasticsearch
- Line 2: bulk_refresh_time (defaults to 30), is the amount of time that a bulk flush is forced
- Line 3: bulk (defaults to False) determines if an event should get added to a bulk queue
@@ -0,0 +1,145 @@
Match/Query Classes
-------------------
ExistsMatch
^^^^^^^^^^^
Checks to see if a specific field exists in a document
.. code-block:: python
:linenos:
from mozdef_util.query_models import ExistsMatch
ExistsMatch("randomfield")
TermMatch
^^^^^^^^^
Checks if a specific field matches the key
.. code-block:: python
:linenos:
from mozdef_util.query_models import TermMatch
TermMatch("details.ip", "127.0.0.1")
TermsMatch
^^^^^^^^^^
Checks if a specific field matches any of the keys
.. code-block:: python
:linenos:
from mozdef_util.query_models import TermsMatch
TermsMatch("details.ip", ["127.0.0.1", "1.2.3.4"])
WildcardMatch
^^^^^^^^^^^^^
Allows regex to be used in looking for documents that a field contains all or part of a key
.. code-block:: python
:linenos:
from mozdef_util.query_models import WildcardMatch
WildcardMatch('summary', 'test*')
PhraseMatch
^^^^^^^^^^^
Checks if a field contains a specific phrase (includes spaces)
.. code-block:: python
:linenos:
from mozdef_util.query_models import PhraseMatch
PhraseMatch('summary', 'test run')
BooleanMatch
^^^^^^^^^^^^
Used to apply specific "matchers" to a query. This will unlikely be used outside of SearchQuery.
.. code-block:: python
:linenos:
from mozdef_util.query_models import ExistsMatch, TermMatch, BooleanMatch
must = [
ExistsMatch('details.ip')
]
must_not = [
TermMatch('_type', 'alert')
]
BooleanMatch(must=must, should=[], must_not=must_not)
MissingMatch
^^^^^^^^^^^^
Checks if a field does not exist in a document
.. code-block:: python
:linenos:
from mozdef_util.query_models import MissingMatch
MissingMatch('summary')
RangeMatch
^^^^^^^^^^
Checks if a field value is within a specific range (mostly used to look for documents in a time frame)
.. code-block:: python
:linenos:
from mozdef_util.query_models import RangeMatch
RangeMatch('utctimestamp', "2016-08-12T21:07:12.316450+00:00", "2016-08-13T21:07:12.316450+00:00")
QueryStringMatch
^^^^^^^^^^^^^^^^
Uses a custom query string to generate the "match" based on (Similar to what you would see in kibana)
.. code-block:: python
:linenos:
from mozdef_util.query_models import QueryStringMatch
QueryStringMatch('summary: test')
Aggregation
^^^^^^^^^^^
Used to aggregate results based on a specific field
.. code-block:: python
:linenos:
from mozdef_util.query_models import Aggregation, SearchQuery, ExistsMatch
search_query = SearchQuery(hours=24)
must = [
ExistsMatch('seenindicator')
]
search_query.add_must(must)
aggr = Aggregation('details.ip')
search_query.add_aggregation(aggr)
results = search_query.execute(es_client, indices=['events','events-previous'])
Oops, something went wrong.

0 comments on commit a0dd2d7

Please sign in to comment.