Revised README.md, added CloudFormation templates and Python code

README.md

@@ -1,7 +1,14 @@
-## Adding Log Subscription Filters for Centralized Logging
+## Amazon CloudWatch Log Centralizer
 
 Centralized logging infrastructure for multiple AWS accounts using CloudFormation and Python
 
 ## License Summary
 
 This sample code is made available under a modified MIT license. See the LICENSE file.
+
+## Overview
+Please see the AWS Architecture Blog (https://aws.amazon.com/blogs/architecture/) article _Stream Amazon CloudWatch Logs to a Centralized Account for Audit and Analysis_ for instructions and more context.
+
+While some AWS customers use the built-in ability to push Amazon CloudWatch Logs directly into Amazon Elasticsearch Service for analysis, others would prefer to move all logs into a centralized Amazon Simple Storage Service (Amazon S3) bucket location for access by custom and third-party tools. Setting up this solution assumes some knowledge of CloudFormation, Python3 and the boto3 AWS SDK.
+
+You will need to have or configure an AWS working account and logging account, an IAM access and secret key for those accounts, and a working environment containing Python and the boto3 SDK. For assistance, see the Getting Started Resource Center at https://aws.amazon.com/getting-started/ and Start Building with SDKs and Tools at https://aws.amazon.com/getting-started/tools-sdks/.

cfn/AddSubscriptionFilter.yml

@@ -0,0 +1,83 @@
+# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+# SPDX-License-Identifier: MIT-0
+
+AWSTemplateFormatVersion:          2010-09-09
+Description:                       Set up role and lambda to automatically add a subscription filter to all new log groups that are created within the account.
+
+Parameters:
+  InfrastructureS3Bucket:
+    Description:                   Name (not ARN) of the S3 Infrastructure bucket containing the lambda code in a zip file (e.g., infrastructure-bucket).
+    MaxLength:                     250
+    MinLength:                     1
+    Type:                          String
+  LambdaFolderAndFile:
+    Description:                   S3 folder location and the file name of the lambda code in a zip file (e.g., lambdas/AddSubscriptionFilter.zip).
+    MaxLength:                     250
+    MinLength:                     1
+    Type:                          String
+
+Resources:
+  # IAM Roles and Policies
+  AddSubscriptionFilterRole:
+    Type:                          AWS::IAM::Role
+    Properties:
+      AssumeRolePolicyDocument:
+        Statement:
+            Effect:                Allow
+            Principal:
+              Service:
+                - lambda.amazonaws.com
+            Action:                'sts:AssumeRole'
+      Path:                        /
+      ManagedPolicyArns:
+        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
+      Policies:
+      - PolicyName:                AddSubcriptionPolicy
+        PolicyDocument:
+          Statement:
+          - Effect:                Allow
+            Action:
+              - logs:*
+              - ssm:GetParameter
+            Resource:             '*'
+          - Action:
+            - s3:GetObject
+            - s3:GetObjectVersion
+            - s3:GetBucketLocation
+            - s3:ListBucket
+            - s3:PutObject
+            - s3:PutObjectAcl
+            - s3:GetBucketVersioning
+            Effect:                Allow
+            Resource:
+            - arn:aws:s3:::<S3 infrastructure-bucket>
+            - arn:aws:s3:::<S3 infrastructure-bucket>/*
+  # Lambda
+  AddSubscriptionLambda:
+    Type:                         AWS::Lambda::Function
+    Properties:
+      Handler:                    lambda.lambda_handler
+      Runtime:                    python3.6
+      Code:
+        S3Bucket: !Ref            InfrastructureS3Bucket
+        S3Key: !Ref               LambdaFolderAndFile
+      Role: !GetAtt               'AddSubscriptionFilterRole.Arn'
+  AddSubscriptionLambdaPermission:    # Grants the event rule permission to invoke the lambda
+    Type:                         AWS::Lambda::Permission
+    Properties:
+      Action:                     lambda:InvokeFunction
+      FunctionName: !GetAtt       'AddSubscriptionLambda.Arn'
+      Principal:                  events.amazonaws.com
+      SourceArn: !GetAtt          'CreateLogGroupEvent.Arn'
+  # CloudWatch event rule
+  CreateLogGroupEvent:
+    Type:                         AWS::Events::Rule
+    Properties:
+      Description:                Event rule for identifying new CloudWatch Logs log groups and calling Lambda to add new subscription filters
+      EventPattern:               '{"source":["aws.logs"],"detail-type":["AWS API Call via CloudTrail"],"detail":{"eventSource":["logs.amazonaws.com"],"eventName":["CreateLogGroup"]}}'
+      Name:                       CreateLogGroupEvent
+      State:                      ENABLED
+      Targets:
+        -
+          Arn:                    !GetAtt 'AddSubscriptionLambda.Arn'
+          Id:                     add_new_subscription

cfn/centralLogging.yml

@@ -0,0 +1,150 @@
+# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+# SPDX-License-Identifier: MIT-0
+
+AWSTemplateFormatVersion:          2010-09-09
+Description:                       Set up roles, logs destination, kinesis and delivery stream for centralizing logs to S3 in centralized logging account
+
+Parameters:
+  LoggingS3Bucket:
+    Description:                   ARN of the S3 Logging bucket to write to.
+    MaxLength:                     250
+    MinLength:                     1
+    Type:                          String
+
+Resources:
+  # IAM Roles
+  CWLtoKinesisRole:
+    Type:                          'AWS::IAM::Role'
+    Properties:
+      AssumeRolePolicyDocument:
+        Version:                   2012-10-17
+        Statement:
+          - Sid:                   ''
+            Effect:                Allow
+            Principal:
+              Service:
+                - logs.us-west-2.amazonaws.com
+                - logs.us-west-1.amazonaws.com
+                - logs.us-east-2.amazonaws.com
+                - logs.us-east-1.amazonaws.com
+                - logs.ap-south-1.amazonaws.com
+                - logs.ap-northeast-2.amazonaws.com
+                - logs.ap-southeast-1.amazonaws.com
+                - logs.ap-southeast-2.amazonaws.com
+                - logs.ap-northeast-1.amazonaws.com
+                - logs.ca-central-1.amazonaws.com
+                - logs.eu-central-1.amazonaws.com
+                - logs.eu-west-1.amazonaws.com
+                - logs.eu-west-2.amazonaws.com
+                - logs.eu-west-3.amazonaws.com
+                - logs.sa-east-1.amazonaws.com
+            Action:                'sts:AssumeRole'
+      Path:                        '/'
+  CWLtoKinesisPolicy:
+    Type:                          'AWS::IAM::Policy'
+    Properties:
+      PolicyName:                  CWL_to_Kinesis_Policy
+      PolicyDocument:
+        Version:                   2012-10-17
+        Statement:
+          - Effect:                Allow
+            Action:
+              - 'kinesis:PutRecord'
+            Resource:
+              - !GetAtt            'KinesisLoggingStream.Arn'
+          - Effect:                Allow
+            Action:
+              - 'iam:PassRole'
+            Resource:
+              - !GetAtt           'CWLtoKinesisRole.Arn'
+      Roles:
+        - !Ref                    CWLtoKinesisRole
+  FirehoseDeliveryRole:
+    Type:                          'AWS::IAM::Role'
+    Properties:
+      AssumeRolePolicyDocument:
+        Version:                   2012-10-17
+        Statement:
+          - Sid:                   ''
+            Effect:                Allow
+            Principal:
+              Service:             firehose.amazonaws.com
+            Action:                'sts:AssumeRole'
+            Condition:
+              StringEquals:
+                'sts:ExternalId':  !Ref 'AWS::AccountId'
+  FirehoseDeliveryPolicy:
+    Type:                          'AWS::IAM::Policy'
+    Properties:
+      PolicyName:                  Firehose_Delivery_Policy
+      PolicyDocument:
+        Version:                   2012-10-17
+        Statement:
+          - Effect:                Allow
+            Action:
+              - 's3:AbortMultipartUpload'
+              - 's3:GetBucketLocation'
+              - 's3:GetObject'
+              - 's3:ListBucket'
+              - 's3:ListBucketMultipartUploads'
+              - 's3:PutObject'
+            Resource:
+              - !Ref LoggingS3Bucket
+              - !Join
+                - ''
+                - - !Ref LoggingS3Bucket
+                  - '*'
+          - Effect:               Allow
+            Action:
+              - 'kinesis:DescribeStream'
+              - 'kinesis:GetShardIterator'
+              - 'kinesis:GetRecords'
+            Resource:             !GetAtt 'KinesisLoggingStream.Arn'
+      Roles:
+        - !Ref FirehoseDeliveryRole
+  # Log Destination
+  LogDestination:
+    Type:                         AWS::Logs::Destination
+    DependsOn:
+      - KinesisLoggingStream
+      - CWLtoKinesisRole
+      - CWLtoKinesisPolicy
+    Properties:
+      DestinationName:            CentralLogDestination
+      RoleArn:                    !GetAtt 'CWLtoKinesisRole.Arn'
+      TargetArn:                  !GetAtt 'KinesisLoggingStream.Arn'
+      DestinationPolicy: "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"AWS\":[\"<AccountNumberHere>\",\"<AccountNumberHere>\",\"<AccountNumberHere>\",\"<AccountNumberHere>\",\"<AccountNumberHere>\"]},\"Action\":\"logs:PutSubscriptionFilter\",\"Resource\":\"arn:aws:logs:<region>:<logging account number>:destination:CentralLogDestination\"}]}"
+  # Kinesis
+  KinesisLoggingStream:
+    Type:                         AWS::Kinesis::Stream
+    Properties:
+      Name:                       'Kinesis-Centralized-Logging-Stream'
+      RetentionPeriodHours:       48
+      ShardCount:                 2
+  # Firehose
+  FirehoseLoggingDeliveryStream:
+    Type:                                 'AWS::KinesisFirehose::DeliveryStream'
+    DependsOn:
+      - KinesisLoggingStream
+      - FirehoseDeliveryRole
+      - FirehoseDeliveryPolicy
+    Properties:
+      DeliveryStreamName:                 'Centralized-Logging-Delivery-Stream'
+      DeliveryStreamType:                  KinesisStreamAsSource
+      KinesisStreamSourceConfiguration:
+        KinesisStreamARN:                 !GetAtt 'KinesisLoggingStream.Arn'
+        RoleARN:                          !GetAtt 'FirehoseDeliveryRole.Arn'
+      ExtendedS3DestinationConfiguration:
+        BucketARN:                        !Ref LoggingS3Bucket
+        BufferingHints:
+          IntervalInSeconds:              '300'
+          SizeInMBs:                      '50'
+        CompressionFormat:                UNCOMPRESSED
+        Prefix:                           'CentralizedAccountLogs/'
+        RoleARN:                          !GetAtt 'FirehoseDeliveryRole.Arn'
+Outputs:
+  DestinationArnExport:
+    Description:                   ARN for the LogDestination
+    Export:
+      Name:                        LogDestinationArn
+    Value:                         !GetAtt 'LogDestination.Arn'

python/CentralLogging.py

@@ -0,0 +1,63 @@
+# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+# SPDX-License-Identifier: MIT-0
+
+import boto3
+
+
+class CentralLogging:
+    def __init__(self):
+        self.log_client = boto3.client('logs')
+        self.ssm_client = boto3.client('ssm')
+        print('Central Logger Made')
+
+    def add_subscriptions_to_existing_log_groups(self):
+
+        # Retrieve all of the existing log groups
+        log_group_response = self.log_client.describe_log_groups()
+
+        # If there are log groups, iterate over each one, retrieve its name, and call code to add the subscription to it
+        if log_group_response:
+
+            for log_group in log_group_response['logGroups']:
+                log_group_name = log_group['logGroupName']
+                self.add_subscription_filter(log_group_name)
+
+    # Add subscription to centralized logging to the log group with log_group_name
+    def add_subscription_filter(self, log_group_name):
+        # Retrieve the destination for the subscription from the Parameter Store
+        destination_response = self.ssm_client.get_parameter(Name='LogDestination')
+
+        # Error if no destination, otherwise extract destination id from response
+        if not destination_response:
+            raise ValueError(
+                'Cannot locate central logging destination, put_subscription_filter failed')
+        else:
+            destination = destination_response['Parameter']['Value']
+
+        # Error to try to add subscription if one already exists, so delete any existing subscription from this log group
+        self.delete_existing_subscription_filter(log_group_name)
+
+        # Put the new subscription with the destination onto the log group
+        self.log_client.put_subscription_filter(
+            logGroupName=log_group_name,
+            filterName='Destination',
+            filterPattern='',
+            destinationArn=destination
+        )
+
+    # Delete an existing subscription from the log group
+    def delete_existing_subscription_filter(self, log_group_name):
+        # Retrieve any existing subscription filters (only can be one)
+        subscription_filters = self.log_client.describe_subscription_filters(
+            logGroupName=log_group_name)
+
+        # Iterate over results if there are any (again, should not be multiple, but to follow the convetion of the SDK)
+        for subscription_filter in subscription_filters['subscriptionFilters']:
+            # Retrieve the subscription filter name to use in the call to delete
+            filter_name = subscription_filter['filterName']
+
+            # Delete any subscriptions that are found on the log group
+            self.log_client.delete_subscription_filter(
+                logGroupName=log_group_name,
+                filterName=filter_name
+            )

python/init_account_central_logging.py

@@ -0,0 +1,43 @@
+#!/usr/bin/env python
+
+# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+# SPDX-License-Identifier: MIT-0
+
+from __future__ import print_function, absolute_import
+import boto3
+from CentralLogging import CentralLogging
+
+
+def add_log(destination_arn):
+    central_logger = CentralLogging()
+    # Initializes the account for centralized logging by adding the logging destination to the Parameter Store and then adding a subscription to the destination for each existing log group
+
+    # Create an SSM SDK client to handle Parameter Store actions
+    ssm_client = boto3.client('ssm')
+
+    # Put the destination (currently hard-coded and needs to be changed if the destination ever changes) into the Parameter Store
+    destination_response = ssm_client.put_parameter(
+        Name='LogDestination',
+        Description='Centralized logging account destination ARN for subscription filters',
+        Value=destination_arn,
+        Type='String',
+        Overwrite=True
+    )
+
+    print(destination_response)
+    # Call the code to add a subscription to all existing log groups
+    central_logger.add_subscriptions_to_existing_log_groups()
+
+
+if __name__ == "__main__":
+    import argparse
+
+    parser = argparse.ArgumentParser()
+    parser.add_argument("-d", "--destination", help="logs:destination ARN to send logs to")
+
+    args = parser.parse_args()
+    if(args.destination):
+        add_log(args.destination)
+    else:
+        parser.print_help()
+        raise ValueError('Must provide LogDestination ARN. See help')

python/lambda.py

@@ -0,0 +1,26 @@
+# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+# SPDX-License-Identifier: MIT-0
+
+import boto3
+from CentralLogging import CentralLogging
+
+# Lambda handler triggered by a CloudWatch Event whenever a new log group is created.  Calls the core code to add new subscription to the newly created log group
+
+
+def lambda_handler(event, context):
+    central_logger = CentralLogging()
+    print(event)
+    print("In add_new_subscription Lambda - requestParameters: ")
+
+    # Retrieves the request parameters from the event that was called to create the log group
+    request_parameters = event['detail']['requestParameters']
+    print(request_parameters)
+
+    # Extract the log group name from the request parameters to create the log group
+    if request_parameters:
+        print("     Inspecting request parameters for log_group_name:")
+        log_group_name = request_parameters['logGroupName']
+        print(log_group_name)
+
+        # Call code to add the subscription to the log group
+        central_logger.add_subscription_filter(log_group_name)