Skip to content
Permalink
Browse files

Merge branch 'master' into geomodel-v-0-1

  • Loading branch information...
arcrose committed Aug 2, 2019
2 parents 74c2092 + e46d4de commit c4e222133edacdd0949ebdeb94eed8c2c808da88
Showing with 742 additions and 294 deletions.
  1. +2 −2 .gitignore
  2. +55 −1 CHANGELOG → CHANGELOG.md
  3. +1 −1 CODEOWNERS
  4. +2 −2 README.md
  5. +3 −0 alerts/ldap_password_spray.conf
  6. +48 −0 alerts/ldap_password_spray.py
  7. +2 −1 alerts/lib/alerttask.py
  8. +6 −2 alerts/plugins/ip_source_enrichment.py
  9. +2 −0 alerts/plugins/ipaddr.conf
  10. +130 −0 alerts/plugins/ipaddr.py
  11. +3 −0 alerts/plugins/port_scan_enrichment.py
  12. +1 −1 alerts/proxy_drop_executable.py
  13. +1 −1 alerts/ssh_access_signreleng.py
  14. +1 −1 alerts/ssh_lateral.py
  15. +1 −1 alerts/unauth_ssh.py
  16. +1 −1 cloudy_mozdef/cloudformation/mozdef-es.yml
  17. +1 −1 config/50-mozdef-filter.conf
  18. +3 −6 cron/auth02mozdef.py
  19. +1 −1 cron/createFDQNBlockList.py
  20. +6 −3 cron/healthToMongo.py
  21. +1 −1 docker/compose/elasticsearch/files/elasticsearch.yml
  22. +3 −3 docker/compose/mozdef_bootstrap/files/initial_setup.py
  23. +13 −0 docker/compose/mozdef_bootstrap/files/resources/cloudtrail_eventname_pie_graph.json
  24. +13 −0 docker/compose/mozdef_bootstrap/files/resources/cloudtrail_eventname_table.json
  25. +15 −0 docker/compose/mozdef_bootstrap/files/resources/cloudtrail_events_dashboard.json
  26. +13 −0 docker/compose/mozdef_bootstrap/files/resources/cloudtrail_events_line_graph.json
  27. +13 −0 docker/compose/mozdef_bootstrap/files/resources/cloudtrail_events_map.json
  28. +13 −0 docker/compose/mozdef_bootstrap/files/resources/cloudtrail_total_event_count.json
  29. +13 −0 docker/compose/mozdef_bootstrap/files/resources/cloudtrail_user_identity_table.json
  30. +1 −1 docs/source/alert_development_guide.rst
  31. +6 −3 docs/source/benchmarking.rst
  32. +26 −26 docs/source/cicd.rst
  33. +8 −7 docs/source/cloud_deployment.rst
  34. +7 −0 docs/source/installation.rst
  35. +18 −18 docs/source/overview.rst
  36. +8 −6 docs/source/usage.rst
  37. +1 −1 meteor/client/themes/side_nav_dark/menu.html
  38. +5 −13 meteor/imports/themes/classic/mozdef.css
  39. +15 −10 meteor/imports/themes/dark/mozdef.css
  40. +13 −16 meteor/imports/themes/light/mozdef.css
  41. +30 −25 meteor/imports/themes/side_nav_dark/mozdef.css
  42. +0 −1 mq/esworker_cloudtrail.py
  43. +0 −1 mq/esworker_sqs.py
  44. +3 −2 mq/plugins/broFixup.py
  45. +3 −1 mq/plugins/cloudtrail.py
  46. +7 −7 mq/plugins/parse_sshd.py
  47. +1 −1 mq/plugins/parse_su.py
  48. +6 −7 rest/index.py
  49. +4 −4 rest/plugins/cymon.py
  50. +30 −30 rest/plugins/fqdnblocklist.py
  51. +32 −32 rest/plugins/ipblocklist.py
  52. +2 −2 rest/plugins/logincounts.py
  53. +17 −17 rest/plugins/vpc_blackhole.py
  54. +22 −23 rest/plugins/watchlist.py
  55. +99 −0 tests/alerts/test_ldap_password_spray.py
  56. +3 −3 tests/mozdef_util/query_models/test_query_string_match.py
  57. +3 −3 tests/mq/plugins/test_broFixup.py
  58. +5 −5 tests/mq/plugins/test_suricataFixup.py
@@ -13,8 +13,8 @@ alerts/generic_alerts
/.project
/data
.vscode
cloudy_mozdef/aws_parameters.json
cloudy_mozdef/aws_parameters.sh
cloudy_mozdef/aws_parameters.*.json
cloudy_mozdef/aws_parameters.*.sh
docs/source/_build
docs/source/_static
*.swp
@@ -5,6 +5,58 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)

## [Unreleased]

## [v3.1.1] - 2019-07-25

### Added
- Ability to get open indices in ElasticsearchClient
- Documentation on installing dependencies on Mac OS X

### Changed
- AWS Managed Elasticsearch/Kibana version to 6.7

### Fixed
- Disk free/total in /about page shows at most 2 decimal places
- Connections to SQS and S3 without access key and secret
- Ability to block IPs and add to Watchlist


## [v3.1.0] - 2019-07-18

### Added
- Captured the AWS CodeBuild CI/CD configuration in code with documentation
- Support for HTTP Basic Auth in AWS deployment
- Docker healthchecks to docker containers
- Descriptions to all AWS Lambda functions
- Support for alerts-* index in docker environment
- Alert that detects excessive numbers of AWS API describe calls
- Additional AWS infrastructure to support AWS re:Inforce 2019 workshop
- Documentation specific to MozDef installation now that MozDef uses Python 3
- Config setting for CloudTrail notification SQS queue polling time
- Config setting for Slack bot welcome message

### Changed
- Kibana port from 9443 to 9090
- AWS CloudFormation default values from "unset" to empty string
- Simplify mozdef-mq logic determining AMQP endpoint URI
- SQS to always use secure transport
- CloudTrail alert unit tests
- Incident summary placeholder text for greater clarity
- Display of Veris data for easier viewing
- All Dockerfiles to reduce image size, pin package signing keys and improve
clarity

### Fixed
- Workers starting before GeoIP data is available
- Mismatched MozDefACMCertArn parameter name in CloudFormation template
- Duplicate mozdefvpcflowlogs object
- Hard coded AWS Availability Zone
- httplib2 by updating to version to 0.13.0 for python3
- mozdef_util by modifying bulk queue to acquire lock before saving events
- Dashboard Kibana URL
- Unnecessary and conflicting package dependencies from MozDef and mozdef_util
- get_indices to include closed indices


## [v3.0.0] - 2019-07-08
### Added
- Support for Python3
@@ -132,7 +184,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
- Added checks on sending SQS messages to only accept intra-account messages
- Improved docker performance and disk space requirements
[Unreleased]: https://github.com/mozilla/MozDef/compare/v3.0.0...HEAD
[Unreleased]: https://github.com/mozilla/MozDef/compare/v3.1.1...HEAD
[v3.1.1]: https://github.com/mozilla/MozDef/compare/v3.1.0...v3.1.1
[v3.1.0]: https://github.com/mozilla/MozDef/compare/v3.0.0...v3.1.0
[v3.0.0]: https://github.com/mozilla/MozDef/compare/v2.0.1...v3.0.0
[v2.0.1]: https://github.com/mozilla/MozDef/compare/v2.0.0...v2.0.1
[v2.0.0]: https://github.com/mozilla/MozDef/compare/v1.40.0...v2.0.0
@@ -8,5 +8,5 @@

# Entire set can review certain documentation files
/README.md @pwnbus @mpurzynski @Phrozyn @tristanweir @gene1wood @andrewkrug
/CHANGELOG @pwnbus @mpurzynski @Phrozyn @tristanweir @gene1wood @andrewkrug
/CHANGELOG.md @pwnbus @mpurzynski @Phrozyn @tristanweir @gene1wood @andrewkrug
/docs/ @pwnbus @mpurzynski @Phrozyn @tristanweir @gene1wood @andrewkrug
@@ -25,7 +25,7 @@ The Mozilla Enterprise Defense Platform (MozDef) seeks to automate the security

## Goals:

* Provide a platform for use by defenders to rapidly discover and respond to security incidents.
* Provide a platform for use by defenders to rapidly discover and respond to security incidents
* Automate interfaces to other systems like bunker, cymon, mig
* Provide metrics for security events and incidents
* Facilitate real-time collaboration amongst incident handlers
@@ -36,7 +36,7 @@ The Mozilla Enterprise Defense Platform (MozDef) seeks to automate the security

MozDef is in production at Mozilla where we are using it to process over 300 million events per day.

[1]: https://console.aws.amazon.com/cloudformation/home?region=us-west-2#/stacks/new?stackName=mozdef-for-aws&templateURL=https://s3-us-west-2.amazonaws.com/public.us-west-2.infosec.mozilla.org/mozdef/cf/v1.38.5/mozdef-parent.yml
[1]: https://console.aws.amazon.com/cloudformation/home?region=us-west-2#/stacks/new?stackName=mozdef-for-aws&templateURL=https://s3-us-west-2.amazonaws.com/public.us-west-2.infosec.mozilla.org/mozdef/cf/v3.1.1/mozdef-parent.yml

## Survey & Contacting us

@@ -0,0 +1,3 @@
[options]
threshold_count = 1
search_depth_min = 60
@@ -0,0 +1,48 @@
#!/usr/bin/env python

# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
# Copyright (c) 2014 Mozilla Corporation


from lib.alerttask import AlertTask
from mozdef_util.query_models import SearchQuery, TermMatch
import re


class AlertLdapPasswordSpray(AlertTask):
def main(self):
self.parse_config('ldap_password_spray.conf', ['threshold_count', 'search_depth_min'])
search_query = SearchQuery(minutes=int(self.config.search_depth_min))
search_query.add_must([
TermMatch('category', 'ldap'),
TermMatch('details.response.error', 'LDAP_INVALID_CREDENTIALS')
])
self.filtersManual(search_query)
self.searchEventsAggregated('details.client', samplesLimit=10)
self.walkAggregations(threshold=int(self.config.threshold_count))

def onAggregation(self, aggreg):
category = 'ldap'
tags = ['ldap']
severity = 'WARNING'
email_list = set()
email_regex = r'.*mail=([a-zA-Z0-9._-]+@[a-zA-Z0-9._-]+\.[a-zA-Z0-9_-]+)'

for event in aggreg['allevents']:
for request in event['_source']['details']['requests']:
match_object = re.match(email_regex, request['details'][0])
if match_object:
email_list.add(match_object.group(1))

# If no emails, don't throw alert
# if len(email_list) == 0:
# return None

summary = 'LDAP Password Spray Attack in Progress from {0} targeting the following account(s): {1}'.format(
aggreg['value'],
",".join(sorted(email_list))
)

return self.createAlertDict(summary, category, tags, aggreg['events'], severity)
@@ -20,6 +20,7 @@
from celery.utils.log import get_task_logger

from mozdef_util.utilities.toUTC import toUTC
from mozdef_util.utilities.logger import logger
from mozdef_util.elasticsearch_client import ElasticsearchClient
from mozdef_util.query_models import TermMatch, ExistsMatch

@@ -545,6 +546,6 @@ def parse_json_alert_config(self, config_file):
try:
json_obj = json.load(fd)
except ValueError:
sys.stderr.write("FAILED to open the configuration file\n")
logger.error("FAILED to open the configuration file\n")

return json_obj
@@ -20,8 +20,8 @@
def _find_ip_addresses(string):
'''List all of the IPv4 and IPv6 addresses found in a string.'''

ipv4_rx = '(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)'
ipv6_rx = '(([0-9a-fA-F]{1,4}:){7,7}[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,7}:|([0-9a-fA-F]{1,4}:){1,6}:[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,5}(:[0-9a-fA-F]{1,4}){1,2}|([0-9a-fA-F]{1,4}:){1,4}(:[0-9a-fA-F]{1,4}){1,3}|([0-9a-fA-F]{1,4}:){1,3}(:[0-9a-fA-F]{1,4}){1,4}|([0-9a-fA-F]{1,4}:){1,2}(:[0-9a-fA-F]{1,4}){1,5}|[0-9a-fA-F]{1,4}:((:[0-9a-fA-F]{1,4}){1,6})|:((:[0-9a-fA-F]{1,4}){1,7}|:)|fe80:(:[0-9a-fA-F]{0,4}){0,4}%[0-9a-zA-Z]{1,}|::(ffff(:0{1,4}){0,1}:){0,1}((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])|([0-9a-fA-F]{1,4}:){1,4}:((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9]))'
ipv4_rx = r'(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)'
ipv6_rx = r'(([0-9a-fA-F]{1,4}:){7,7}[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,7}:|([0-9a-fA-F]{1,4}:){1,6}:[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,5}(:[0-9a-fA-F]{1,4}){1,2}|([0-9a-fA-F]{1,4}:){1,4}(:[0-9a-fA-F]{1,4}){1,3}|([0-9a-fA-F]{1,4}:){1,3}(:[0-9a-fA-F]{1,4}){1,4}|([0-9a-fA-F]{1,4}:){1,2}(:[0-9a-fA-F]{1,4}){1,5}|[0-9a-fA-F]{1,4}:((:[0-9a-fA-F]{1,4}){1,6})|:((:[0-9a-fA-F]{1,4}){1,7}|:)|fe80:(:[0-9a-fA-F]{0,4}){0,4}%[0-9a-zA-Z]{1,}|::(ffff(:0{1,4}){0,1}:){0,1}((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])|([0-9a-fA-F]{1,4}:){1,4}:((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9]))'

ipv4 = re.findall(ipv4_rx, string)
ipv6_map = map(
@@ -59,6 +59,8 @@ def ip_in_range(ip):

alert = alert.copy()

if 'details' not in alert:
alert['details'] = {}
alert['details']['sites'] = []

for ip in set(ips):
@@ -140,6 +142,8 @@ class message(object):
'''

def __init__(self):
# Run plugin on all alerts
self.registration = '*'
self._config = _load_config(CONFIG_FILE)

def onMessage(self, message):
@@ -0,0 +1,2 @@
[options]
keywords =
@@ -0,0 +1,130 @@
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
# Copyright (c) 2014 Mozilla Corporation

from configlib import getConfig, OptionParser
import netaddr
import os


def isIPv4(ip):
try:
return netaddr.valid_ipv4(ip)
except:
return False


def isIPv6(ip):
try:
return netaddr.valid_ipv6(ip)
except:
return False


def addError(message, error):
'''add an error note to a message'''
if 'errors' not in message:
message['errors'] = list()
if isinstance(message['errors'], list):
message['errors'].append(error)


class message(object):
def __init__(self):
'''
uses heuristic to find and attach the source IP address of the alert
'''

# set my own conf file
# relative path to the rest index.py file
self.configfile = os.path.join(os.path.dirname(__file__), 'ipaddr.conf')
self.options = None
if os.path.exists(self.configfile):
self.initConfiguration()

self.registration = self.options.keywords.split(" ")
self.priority = 1

def initConfiguration(self):
myparser = OptionParser()
# setup self.options by sending empty list [] to parse_args
(self.options, args) = myparser.parse_args([])

# fill self.options with plugin-specific options
self.options.keywords = getConfig('keywords', 'localhost', self.configfile)

def onMessage(self, message):
'''
Examine possible ip addresses for the following:
ipv6 in an ipv4 field
ipv4 in another field
'-' or other invalid ip in the ip field
Also sets ipv4 in two fields:
ipaddress (decimal mapping IP)
ipv4address (string mapping)
Elastic search is inconsistent about returning IPs as
decimal or IPs.
In a query an IP field is returned as string.
In a facets an IP field is returned as decimal.
No ES field type exists for ipv6, so always having
a string version is the most flexible option.
'''

# here is where you do something with the incoming alert message
if 'events' in message:
if 'documentsource' in message['events'][0]:
if 'details' in message['events'][0]['documentsource']:
event = message['events'][0]['documentsource']['details']
if 'details' not in message:
message['details'] = {}
# forwarded header can be spoofed, so try it first,
# but override later if we've a better field.
if 'http_x_forwarded_for' in event:
# should be a comma delimited list of ips with the original client listed first
ipText = event['http_x_forwarded_for'].split(',')[0]
if isIPv4(ipText) and 'sourceipaddress' not in event:
message['details']['sourceipaddress'] = ipText
if isIPv4(ipText) and 'sourceipv4address' not in event:
message['details']['sourceipv4address'] = ipText
if isIPv6(ipText) and 'sourceipv6address' not in event:
message['details']['sourceipv6address'] = ipText

if 'sourceipaddress' in event:
ipText = event['sourceipaddress']
if isIPv6(ipText):
event['sourceipv6address'] = ipText
message['details']['sourceipaddress'] = '0.0.0.0'
addError(message, 'plugin: {0} error: {1}'.format('ipFixUp.py', 'sourceipaddress is ipv6, moved'))
elif isIPv4(ipText):
message['details']['sourceipv4address'] = ipText
message['details']['sourceipaddress'] = ipText
else:
# Smells like a hostname, let's save it as source field
message['details']['source'] = event['sourceipaddress']
message['details']['sourceipaddress'] = None

if 'destinationipaddress' in event:
ipText = event['destinationipaddress']
if isIPv6(ipText):
message['details']['destinationipv6address'] = ipText
message['details']['destinationipaddress'] = '0.0.0.0'
addError(message, 'plugin: {0} error: {1}'.format('ipFixUp.py', 'destinationipaddress is ipv6, moved'))
elif isIPv4(ipText):
message['details']['destinationipv4address'] = ipText
message['details']['destinationipaddress'] = ipText
else:
# Smells like a hostname, let's save it as destination field
message['details']['destination'] = event['destinationipaddress']
message['details']['destinationipaddress'] = None

if 'cluster_client_ip' in event:
ipText = event['cluster_client_ip']
if isIPv4(ipText):
message['details']['sourceipaddress'] = ipText

# you can modify the message if needed
# plugins registered with lower (>2) priority
# will receive the message and can also act on it
# but even if not modified, you must return it
return message
@@ -88,6 +88,9 @@ class message(object):
'''

def __init__(self):
# Run plugin on portscan alerts
self.registration = 'portscan'

config = _load_config(CONFIG_FILE)

try:
@@ -24,7 +24,7 @@ def main(self):
)

# Only notify on certain file extensions from config
filename_regex = "/.*\.({0})/".format(self.config.extensions.replace(",", "|"))
filename_regex = r"/.*\.({0})/".format(self.config.extensions.replace(",", "|"))
search_query.add_must(
[QueryStringMatch("details.destination: {}".format(filename_regex))]
)
@@ -57,7 +57,7 @@ def onEvent(self, event):
sourceipaddress = x['details']['sourceipaddress']

targetuser = 'unknown'
expr = re.compile('Accepted publickey for ([A-Za-z0-9]+) from')
expr = re.compile(r'Accepted publickey for ([A-Za-z0-9]+) from')
m = expr.match(event['_source']['summary'])
groups = m.groups()
if len(groups) > 0:
@@ -124,7 +124,7 @@ def onAggregation(self, aggreg):
source_ips = []
users = []
for x in aggreg['events']:
m = re.match('Accepted publickey for (\S+) from (\S+).*', x['_source']['summary'])
m = re.match(r'Accepted publickey for (\S+) from (\S+).*', x['_source']['summary'])
if m is not None and len(m.groups()) == 2:
ipaddr = netaddr.IPAddress(m.group(2))
for y in self._config['alertifsource']:
@@ -65,7 +65,7 @@ def onEvent(self, event):
sourceipaddress = x['details']['sourceipaddress']

targetuser = 'unknown'
expr = re.compile('Accepted publickey for ([A-Za-z0-9@.\-]+) from')
expr = re.compile(r'Accepted publickey for ([A-Za-z0-9@.\-]+) from')
m = expr.match(event['_source']['summary'])
groups = m.groups()
if len(groups) > 0:
@@ -63,7 +63,7 @@ Resources:
EBSEnabled: true
VolumeType: gp2
VolumeSize: !Ref BlockStoreSizeGB
ElasticsearchVersion: '5.6'
ElasticsearchVersion: '6.7'
ElasticsearchClusterConfig:
InstanceCount: !Ref ESInstanceCount
AccessPolicies:

0 comments on commit c4e2221

Please sign in to comment.
You can’t perform that action at this time.