Skip to content

Commit

Permalink
[terraform] update cloudwatch and flow logs terraform module to reduc…
Browse files Browse the repository at this point in the history 
…e redundancy (#1041)

* adding aliased terraform providers for regions

* updating tf_flow_logs module to remove redundancy

* adding a tf_cloudwatch_logs_destination module to replace tf_cloudwatch module

* removing replaced tf_cloudwatch module

* updating infinitedict function to support initial value

* updating terraform generation code for aforementioned changes

* updating tests 01

* removing legacy test because we ain't living in the past

* fixing tests for default terraform settings

* updating tests 02

* updates to documentation

* addressing PR feedback
  • Loading branch information
@ryandeivert
ryandeivert committed Oct 18, 2019
1 parent 15e06c1 commit 92fdf71
Show file tree
Hide file tree
Showing 31 changed files with 665 additions and 641 deletions.
50 changes: 21 additions & 29 deletions docs/source/clusters.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -143,9 +143,6 @@ Example: CloudTrail via CloudWatch Logs
{
"id": "cloudtrail-via-cloudwatch",
"modules": {
"cloudwatch": {
"enabled": true
},
"cloudtrail": {
"enable_kinesis": true,
"enable_logging": true,
Expand Down Expand Up @@ -175,9 +172,9 @@ Example: CloudTrail via CloudWatch Logs
}
This also creates the CloudTrail and S3 bucket, but now the CloudTrail logs are also delivered to
CloudWatch Logs and then to a Kinesis subscription which feeds the classifier function. This can scale to
higher throughput, since StreamAlert does not have to download potentially very large files from
S3. In this case, rules should be written against the ``cloudwatch:events`` log type.
CloudWatch Logs and then to a Kinesis stream via a CloudWatch Logs Subscription Filter.
This can scale to higher throughput, since StreamAlert does not have to download potentially very
large files from S3. In this case, rules should be written against the ``cloudwatch:cloudtrail`` log type.

Configuration Options
~~~~~~~~~~~~~~~~~~~~~
Expand Down Expand Up @@ -207,7 +204,7 @@ from any AWS account. A common use case is to ingest and scan CloudTrail from mu

.. note:: The :ref:`Kinesis module <kinesis_module>` must also be enabled.

This module is implemented by `terraform/modules/tf_cloudwatch <https://github.com/airbnb/streamalert/tree/stable/terraform/modules/tf_cloudwatch>`_.
This module is implemented by `terraform/modules/tf_cloudwatch_logs_destination <https://github.com/airbnb/streamalert/tree/stable/terraform/modules/tf_cloudwatch_logs_destination>`_.

Example: CloudWatch Logs Cluster
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Expand All @@ -216,16 +213,14 @@ Example: CloudWatch Logs Cluster
{
"id": "cloudwatch-logs-example",
"modules": {
"cloudwatch": {
"cloudwatch_logs_destination": {
"cross_account_ids": [
"111111111111"
],
"enabled": true,
"excluded_regions": [
"regions": [
"ap-northeast-1",
"ap-northeast-2",
"ap-south-1",
"ap-southeast-1",
"ap-southeast-2"
]
},
Expand Down Expand Up @@ -546,12 +541,8 @@ Example: Flow Logs Cluster
"id": "prod",
"modules": {
"flow_logs": {
"cross_account_ids": [
"111111111111"
],
"enis": [],
"enabled": true,
"log_group_name": "flow-logs-test",
"subnets": [
"subnet-12345678"
],
Expand Down Expand Up @@ -582,24 +573,25 @@ Example: Flow Logs Cluster
"region": "us-east-1"
}
This creates the ``flow-logs-test`` CloudWatch Log group, adds flow logs to the specified subnet
and vpc IDs with the log group as their target, and adds a Kinesis subscription to that log group
for StreamAlert consumption.
This creates the ``<prefix>_prod_streamalert_flow_logs`` CloudWatch Log Group, adds flow logs
to the specified subnet, eni, and vpc IDs with the log group as their target, and adds a CloudWatch
Logs Subscription Filter to that log group to send to Kinesis for consumption by StreamAlert.

Configuration Options
~~~~~~~~~~~~~~~~~~~~~

===================== ============================================ ===============
**Key** **Default** **Description**
--------------------- -------------------------------------------- ---------------
``cross_account_ids`` ``[]`` Authorize flow log delivery from these accounts
``enabled`` --- Toggle flow log creation
``enis`` ``[]`` Add flow logs for these ENIs
``log_group_name`` ``<prefix>_<cluster>_streamalert_flow_logs`` Flow logs are directed to this log group
``subnets`` ``[]`` Add flow logs for these VPC subnet IDs
``vpcs`` ``[]`` Add flow logs for these VPC IDs
===================== ============================================ ===============

===================== ============================================================================================================================================= ===============
**Key** **Default** **Description**
--------------------- --------------------------------------------------------------------------------------------------------------------------------------------- ---------------
``enabled`` --- Toggle flow log creation
``flow_log_filter`` ``[version, account, eni, source, destination, srcport, destport, protocol, packets, bytes, windowstart, windowend, action, flowlogstatus]`` Toggle flow log creation
``log_retention`` ``7`` Day for which logs should be retained in the log group
``enis`` ``[]`` Add flow logs for these ENIs
``subnets`` ``[]`` Add flow logs for these VPC subnet IDs
``vpcs`` ``[]`` Add flow logs for these VPC IDs
===================== ============================================================================================================================================= ===============

.. note:: One of the following **must** be set for this module to have any result: ``enis``, ``subnets``, or ``vpcs``

.. _s3_events:

Expand Down
32 changes: 11 additions & 21 deletions streamalert_cli/terraform/cloudtrail.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
import json

from streamalert.shared.logger import get_logger
from streamalert_cli.terraform.cloudwatch import generate_cloudwatch_destinations_internal

LOGGER = get_logger(__name__)

Expand All @@ -34,26 +35,13 @@ def generate_cloudtrail(cluster_name, cluster_dict, config):
modules = config['clusters'][cluster_name]['modules']
cloudtrail_module = 'cloudtrail_{}'.format(cluster_name)

enabled_legacy = modules['cloudtrail'].get('enabled')

cloudtrail_enabled = modules['cloudtrail'].get('enable_logging', True)
kinesis_enabled = modules['cloudtrail'].get('enable_kinesis', True)
send_to_cloudwatch = modules['cloudtrail'].get('send_to_cloudwatch', False)
exclude_home_region = modules['cloudtrail'].get('exclude_home_region_events', False)

account_ids = list(
set([config['global']['account']['aws_account_id']] + modules['cloudtrail'].get(
'cross_account_ids', [])))

# Allow for backwards compatibility
if enabled_legacy:
del config['clusters'][cluster_name]['modules']['cloudtrail']['enabled']
config['clusters'][cluster_name]['modules']['cloudtrail']['enable_logging'] = True
config['clusters'][cluster_name]['modules']['cloudtrail']['enable_kinesis'] = True
LOGGER.info('Converting legacy CloudTrail config')
config.write()
kinesis_enabled = True
cloudtrail_enabled = True
account_ids = set(modules['cloudtrail'].get('cross_account_ids', []))
account_ids.add(config['global']['account']['aws_account_id'])

existing_trail = modules['cloudtrail'].get('existing_trail', False)
is_global_trail = modules['cloudtrail'].get('is_global_trail', True)
Expand All @@ -73,7 +61,7 @@ def generate_cloudtrail(cluster_name, cluster_dict, config):
module_info = {
'source': 'modules/tf_cloudtrail',
'primary_account_id': config['global']['account']['aws_account_id'],
'account_ids': account_ids,
'account_ids': sorted(account_ids),
'cluster': cluster_name,
'prefix': config['global']['account']['prefix'],
'enable_logging': cloudtrail_enabled,
Expand All @@ -92,11 +80,13 @@ def generate_cloudtrail(cluster_name, cluster_dict, config):
module_info['event_pattern'] = json.dumps(event_pattern)

if send_to_cloudwatch:
destination_arn = modules['cloudtrail'].get(
'cloudwatch_destination_arn',
'${{module.cloudwatch_{}_{}.cloudwatch_destination_arn}}'.format(cluster_name,
region)
)
destination_arn = modules['cloudtrail'].get('cloudwatch_destination_arn')
if not destination_arn:
fmt = '${{module.cloudwatch_logs_destination_{}_{}.cloudwatch_logs_destination_arn}}'
destination_arn = fmt.format(cluster_name, region)
if not generate_cloudwatch_destinations_internal(cluster_name, cluster_dict, config):
return False

module_info['cloudwatch_destination_arn'] = destination_arn

cluster_dict['module'][cloudtrail_module] = module_info
Expand Down
136 changes: 98 additions & 38 deletions streamalert_cli/terraform/cloudwatch.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,63 +18,123 @@
LOGGER = get_logger(__name__)


def generate_cloudwatch(cluster_name, cluster_dict, config):
"""Add the CloudWatch destinations, mapping to the configured kinesis stream
def generate_cloudwatch_destinations(cluster_name, cluster_dict, config):
"""Add any CloudWatch Logs destinations for explicitly specified regions
These configuration options will be merged with any other options set
for use with various other modules that utilize CloudWatch Logs destinations:
tf_flow_logs
tf_cloudtrail
Args:
cluster_name (str): The name of the current cluster being generated
cluster_dict (defaultdict): The dict containing all Terraform config for
a given cluster.
config (dict): The loaded config from the 'conf/' directory
Returns:
bool: True if this module was applied successfully, False otherwise
"""
cloudwatch_module = config['clusters'][cluster_name]['modules']['cloudwatch_logs_destination']
if not cloudwatch_module.get('enabled'):
LOGGER.debug('CloudWatch destinations module is not enabled')
return True # not an error

account_ids = cloudwatch_module.get('cross_account_ids', [])
regions = cloudwatch_module.get('regions')
if not regions:
LOGGER.error(
'CloudWatch destinations must be enabled for at '
'least one region in the \'%s\' cluster',
cluster_name
)
return False

return _generate(cluster_name, cluster_dict, config, account_ids, regions)


def generate_cloudwatch_destinations_internal(cluster_name, cluster_dict, config):
"""Add any CloudWatch Logs destinations needed for internal usage (non-cross-account)
This is currently used to configure additional settings for the following modules:
tf_flow_logs
tf_cloudtrail
These configuration options will be merged with any other options set for use in
the tf_cloudwatch_logs_destination module.
Args:
cluster_name (str): The name of the currently generating cluster
cluster_name (str): The name of the current cluster being generated
cluster_dict (defaultdict): The dict containing all Terraform config for
a given cluster.
a given cluster.
config (dict): The loaded config from the 'conf/' directory
Returns:
bool: Result of applying the cloudwatch module
bool: True if this module was applied successfully, False otherwise
"""
cloudwatch_module = config['clusters'][cluster_name]['modules']['cloudwatch']
account_ids = [config['global']['account']['aws_account_id']]
regions = [config['global']['account']['region']]
return _generate(cluster_name, cluster_dict, config, account_ids, regions)


def _generate(cluster_name, cluster_dict, config, account_ids, regions):
"""Add the CloudWatch destinations, mapping to the configured kinesis stream
if not cloudwatch_module.get('enabled', True):
LOGGER.info('The \'cloudwatch\' module is not enabled, nothing to do.')
return True
Args:
cluster_name (str): The name of the current cluster being generated
cluster_dict (defaultdict): The dict containing all Terraform config for
a given cluster.
config (dict): The loaded config from the 'conf/' directory
Returns:
bool: True if this module was applied successfully, False otherwise
"""
# Ensure that the kinesis module is enabled for this cluster since the
# cloudwatch module will utilize the created stream for sending data
if not config['clusters'][cluster_name]['modules'].get('kinesis'):
LOGGER.error('The \'kinesis\' module must be enabled to enable the '
'\'cloudwatch\' module.')
return False

account_id = config['global']['account']['aws_account_id']
parent_module_name = 'cloudwatch_logs_destination_{}'.format(cluster_name)

prefix = config['global']['account']['prefix']
cross_account_ids = cloudwatch_module.get('cross_account_ids', []) + [account_id]
excluded_regions = set(cloudwatch_module.get('excluded_regions', set()))

# Exclude any desired regions from the entire list of regions
regions = {
'ap-northeast-1',
'ap-northeast-2',
'ap-south-1',
'ap-southeast-1',
'ap-southeast-2',
'ca-central-1',
'eu-central-1',
'eu-west-1',
'eu-west-2',
'eu-west-3',
'sa-east-1',
'us-east-1',
'us-east-2',
'us-west-1',
'us-west-2',
}.difference(excluded_regions)

for region in regions:
cluster_dict['module']['cloudwatch_{}_{}'.format(cluster_name, region)] = {
'source': 'modules/tf_cloudwatch',
'region': region,
'cross_account_ids': cross_account_ids,
stream_arn = '${{module.kinesis_{}.arn}}'.format(cluster_name)

# Merge these regions with any that are already in the configuration
all_regions = sorted(
set(cluster_dict['module'][parent_module_name].get('regions', [])).union(set(regions))
)

cluster_dict['module'][parent_module_name] = {
'source': 'modules/tf_cloudwatch_logs_destination',
'prefix': prefix,
'cluster': cluster_name,
'regions': all_regions,
'destination_kinesis_stream_arn': stream_arn,
}

for region in all_regions:
module_name = 'cloudwatch_logs_destination_{}_{}'.format(cluster_name, region)

# Merge these account IDs with any that are already in the configuration
all_account_ids = set(
cluster_dict['module'][module_name].get('account_ids', [])
).union(set(account_ids))

cluster_dict['module'][module_name] = {
'source': 'modules/tf_cloudwatch_logs_destination/modules/destination',
'prefix': prefix,
'cluster': cluster_name,
'kinesis_stream_arn': '${{module.kinesis_{}.arn}}'.format(cluster_name)
'account_ids': sorted(all_account_ids),
'destination_kinesis_stream_arn': stream_arn,
'cloudwatch_logs_subscription_role_arn': (
'${{module.{}.cloudwatch_logs_subscription_role_arn}}'.format(parent_module_name)
),
'providers': {
# use the aliased provider for this region from providers.tf
'aws': 'aws.{}'.format(region)
},
}

return True
11 changes: 9 additions & 2 deletions streamalert_cli/terraform/common.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,16 @@ class InvalidClusterName(Exception):
"""Exception for invalid cluster names"""


def infinitedict():
def infinitedict(initial_value=None):
"""Create arbitrary levels of dictionary key/values"""
return defaultdict(infinitedict)
initial_value = initial_value or {}

# Recursively cast any subdictionary entries in the initial value to infinitedicts
for key, value in initial_value.items():
if isinstance(value, dict):
initial_value[key] = infinitedict(value)

return defaultdict(infinitedict, initial_value)


def monitoring_topic_name(config):
Expand Down

0 comments on commit 92fdf71

Please sign in to comment.