Alerts not firing

[darakian]

Hey all,

Maybe it's the holiday gremlins that got to me, but I've updated my docker instance to 1.35 and I can't get alerts working again. I'm down to the following very simple alert which is not working

#!/usr/bin/env python

# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
# Copyright (c) 2014 Mozilla Corporation


from lib.alerttask import AlertTask
from mozdef_util.query_models import SearchQuery, TermMatch, QueryStringMatch, ExistsMatch, PhraseMatch, WildcardMatch


class AlertHelloworld(AlertTask):
    def main(self):
        # Create a query to look back the last 20 minutes
        search_query = SearchQuery(minutes=20)

        # Add search terms to our query
        search_query.add_must([
            TermMatch('category', 'helloworld'),
       #     ExistsMatch('details.sourceipaddress'),
        ])

        self.filtersManual(search_query)
        # Search aggregations on field 'sourceipaddress'
        # keep X samples of events at most
        self.searchEventsAggregated('details.sourceipaddress', samplesLimit=10)
        # alert when >= X matching events in an aggregation
        self.walkAggregations(threshold=1)

    # Set alert properties
    def onAggregation(self, aggreg):
        # aggreg['count']: number of items in the aggregation, ex: number of failed login attempts
        # aggreg['value']: value of the aggregation field, ex: toto@example.com
        # aggreg['events']: list of events in the aggregation
        category = 'hellocategory'
        tags = ['hello', 'world']
        severity = 'WARNING'
        summary = "My first alert!"

        # Create the alert object based on these properties
        return self.createAlertDict(summary, category, tags, aggreg['events'], severity)

with

'helloworld.AlertHelloworld': {'schedule': crontab(minute='*/1')},

in my docker/compose/mozdef_alerts/files/config.py. I'm sending simple messages to my instance with curl
curl -v --header "Content-Type: application/json" --request POST --data '{"tags": ["test"],"category": "helloworld"}' http://localhost:8080/events and they are showing up in Kibana.

Is there something obvious that I'm doing wrong? Also is there some way outside of the UI that lets me determine if an alert is working?

[jeffbryner]

ExistsMatch('details.sourceipaddress') means that the fields details.sourceipaddress has to exist in the json data structure (it can be any value.. but it has to be present) before it will match.

So for it to match you'd have to change:

curl -v --header "Content-Type: application/json" --request POST --data '{"tags": ["test"],"category": "helloworld"}' http://localhost:8080/events

to

curl -v --header "Content-Type: application/json" --request POST --data '{"tags": ["test"],"category": "helloworld","details":{"sourceipaddress":"1.2.3.4"}}' http://localhost:8080/events

or remove that from the selection criteria.

Hope that helps!

[darakian]

Sorry, I've commented the sourceip line out. Updated code to reflect that.

[jeffbryner]

Oh, I should add that doing an aggregated search also introduces an 'exists' criteria in the search as well behind the scenes, so even if you remove it from the search_query.add_must, it'll get added back in to satisfy the aggregation here: https://github.com/mozilla/MozDef/blob/master/alerts/lib/alerttask.py#L280

[darakian]

Ahhh I see. So if I aggregate on category that should remove the exists requirement? Also do I need to restart mozdef to propagate the change?

[jeffbryner]

If you don't really care about aggregation and just want it to fire on witnessing one event you can use this pattern that the cloudtrail disabled uses: https://github.com/mozilla/MozDef/blob/master/alerts/cloudtrail_logging_disabled.py#L16

Then you'll get an alert for each event that matches.

And yes if you change the code for an alert; you'd need to restart the alerts process for it to take effect.

[darakian]

Current alert form is below. Also, is there a better way to restart the alerts service with the docker containers than make stop; make run? Still nothing.

#!/usr/bin/env python

# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
# Copyright (c) 2014 Mozilla Corporation


from lib.alerttask import AlertTask
from mozdef_util.query_models import SearchQuery, TermMatch, QueryStringMatch, ExistsMatch, PhraseMatch, WildcardMatch


class AlertHelloworld(AlertTask):
    def main(self):
        # Create a query to look back the last 20 minutes
        search_query = SearchQuery(minutes=20)

        # Add search terms to our query
        search_query.add_must([
            TermMatch('category', 'helloworld'),
        #    ExistsMatch('details.sourceipaddress'),
        ])

        self.filtersManual(search_query)
        # Search aggregations on field 'sourceipaddress'
        # keep X samples of events at most
        self.searchEventsAggregated('category', samplesLimit=10)
        # alert when >= X matching events in an aggregation
        self.walkAggregations(threshold=1)

    # Set alert properties
    def onAggregation(self, aggreg):
        # aggreg['count']: number of items in the aggregation, ex: number of failed login attempts
        # aggreg['value']: value of the aggregation field, ex: toto@example.com
        # aggreg['events']: list of events in the aggregation
        category = 'hellocategory'
        tags = ['hello', 'world']
        severity = 'WARNING'
        summary = "My first alert!"

        # Create the alert object based on these properties
        return self.createAlertDict(summary, category, tags, aggreg['events'], severity)

[jeffbryner]

Looks ok, do you have it scheduled to run ala:
https://github.com/mozilla/MozDef/blob/a4ebe1105b301132b5268d6e3bc70a6f95e85fc9/docs/source/alert_development_guide.rst#scheduling-your-alert

As for restarting. If you are doing active development; I'd just docker exec -it <container_name> bash and do restarts from there.

[darakian]

Ya I've scheduled it. Here's my full alerts schedule with some pseudoonomization

ALERTS = {
    'foo1.AlertFoo1': {'schedule': crontab(minute='*/1')},
    'foo2.AlertFoo2': {'schedule': crontab(minute='*/1')},
    'foo3.AlertFoo3': {'schedule': crontab(minute='*/1')},
    'foo4.AlertFoo4': {'schedule': crontab(minute='*/1')},
    'foo5.AlertFoo5': {'schedule': crontab(minute='*/1')},
    'helloworld.AlertHelloworld': {'schedule': crontab(minute='*/1')},
}

[jeffbryner]

hrm.. lemme spin up a local docker version and see what I run into

[darakian]

Cool. Is it possible that one of my alerts might have an error and somehow kill the alerting process?

Edit:
If I wanted to make a web UI alert creation page. Where in the codebase should that be done? meteor/client/*?

[jeffbryner]

I fired up the docker images and got it to work. I don't think there's anything wrong with your code, it's just strangeness with docker that we are dealing with and how to best use it to develop alerts.

The trick for me was making an overload yml file to let me manually start the alerts, something like :

---
version: '3.7'
services:
  alerts:
    image: mozdef/mozdef_alerts
    build:
      context: ../../
      dockerfile: docker/compose/mozdef_alerts/Dockerfile
      cache_from:
        - mozdef/mozdef_alerts
        - mozdef_alerts:latest
    restart: always
    command: bash -c 'python -i'
    stdin_open: true
    tty: true
    depends_on:
      - base
      - elasticsearch
      - rabbitmq
      - bootstrap
    networks:
      - default

then docker exec -it into the container and run:

source /opt/mozdef/envs/python/bin/activate && celery -A celeryconfig worker --loglevel=info --beat

manually when the alert and config file was ready. I used your curl command above to send an event and when a minute passed it ran and created the alert.

Lemme play around with the best way to offer an 'alert creation docker image/overlay' file and I'll add one to the project.

Yeah, a web UI to at least schedule alerts would be a welcome addition. Yup client code goes in meteor/client but usually involves some server side code for the collection, etc.

[darakian]

Just so I'm clear this fix would create a new yml file in docker/compose/mozdef_alerts/ then docker exec -it (get a shell?) into the mozdef_alerts container? Sorry, I'm still fairly new to docker.

Looking at docker ps it seems that my alerts container is in a reboot loop.
# mozdef/mozdef_alerts "bash -c 'while ! ti…" 2 days ago Restarting (1) 29 seconds ago

Found a missing end quote in one of my alerts which was causing the reboot loop. Fixing this does not enable alerts for me, but I made a PR to try-except on alerts importing anyway.

Edit:
This may have fixed my alerting issue after all, though it took well over a minute for the alerting to come back. Thank you much for the help @jeffbryner and sorry for sending you down a rabbit hole.