Fixes #207 by allowing CloudWatch Logs & S3 to invoke the Forwarder

[philhoff-edeka]

What does this PR do?

fixes #207

Motivation

We cannot add a separate (or update an existing) SubscriptionFilter in order to forward logs from API Gateway etc. to trigger the Forwarder lambda using a CloudWatch Log Group.

Error message in the AWS Console:

Could not execute the lambda function. Make sure you have given CloudWatch Logs permission to execute your function.

Additional Notes

This PR allows CloudWatch Logs and S3 to trigger the Forwarder lambda.

Checklist

• Member of the datadog team has run integration tests

[tianchu]

Hi @philhoff-edeka, thanks for your PR! Did you follow these steps to set up triggers? Normally when you set up a trigger through Lambda console, AWS will take care of the "invoke" permission for you. If you let Datadog manage triggers automatically for you, Datadog will automatically add "invoke" permission to the specific resource generating logs. Generally speaking, we shouldn't need to set up these invoke permissions in the template, and they are very broad (allowing all s3 buckets and log groups to invoke, though not sure if that's a super big deal).

It's actually first time we heard this kind of issue, do you mind posting some steps for us to reproduce? Perhaps we missed something?

[gruebel]

Hi @tianchu, the problem is, when you deploy the Log Forwarder stack and try to add a SubscriptionFilter or Bucket notification trigger via CloudFormation in an other stack you will get the above error.

How to reproduce:

1. Deploy the Log Forwarder Stack
2. Deploy a separate Stack with a LogGroup and SubscriptionFilter

  LogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: /test
  SubscriptionFilter:
    Type: AWS::Logs::SubscriptionFilter
    Properties:
      DestinationArn: !Sub 'arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:DatadogLogForwarder'
      FilterPattern: ''
      LogGroupName: !Ref 'LogGroup'

Datadog can't manage triggers for API Gateway, Batch & custom log groups. We definitely don't want to add the triggers manually through the Lambda Console.

[tianchu]

I see, this makes perfect sense now!