AWS::Events::Rule targetting Cloudwatch logs

[ristkari]

1. Delivering events to Cloudwatch logs with Cloudformation

Interesting discovery with Cloudformation and Eventbridge. As a very simple way of reproducing the problem here is a template snippet. Consider the following:

 CFNLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      RetentionInDays: 3
      LogGroupName: '/aws/events/CFN'

  CFNRule:
    Type: AWS::Events::Rule
    Properties:
      EventPattern:
        source:
          - "aws.states"
      Targets:
        - Id: 'CloudwatchLogsTarget'
          Arn: !Sub "arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:${CFNLogGroup}"
        - Id: 'LambdaTarget'
          Arn: !GetAtt EventsFunction.Arn
          
  EventbridgePermission:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName: !GetAtt EventsFunction.Arn
      Action: lambda:InvokeFunction
      Principal: events.amazonaws.com
      SourceArn: !GetAtt CFNRule.Arn

This is very straight forward way to tell that you would like to route all events from Step Functions to both Cloudwatch logs and also to custom lambda-function. Note the "AWS::Lambda::Permission" that is needed in order to invoke Lambda function. In other words the target needs to have resource policy that allows Eventbridge service to deliver the events.

This works partially. Lambda will get triggered but nothing is delivered to Cloudwatch logs. If you create this via UI it works because console does some magic behind the scenes. The magic in this case is the resource based policy for Cloudwatch.
This is told also on documentation: https://docs.aws.amazon.com/eventbridge/latest/userguide/resource-based-policies-eventbridge.html

"If you use the AWS Management Console to add CloudWatch Logs as the target of a rule, this policy is created automatically. If you use the AWS CLI to add the target, you must create this policy if it doesn't exist."

There is no way to add resource based policies for cloudwatch via cloudformation, you are forced to create custom resource if you want to do it. For Lambda it works because you can create AWS::Lambda::Permission via Cloudformation. Cloudwatch resource policy you cannot. Only way of creating those is via CLI, API or Consoles 'behind the scenes' magic.

So the question is whether there is upcoming support to natively doing this? If you are trying to automate this it's either custom resource for Cloudformation which introduces additional complexity since you have to create your custom resource in different stack. Other option is to use CLi but then your pipeline/automation process is littered with CLI here, cloudformation there - not very hygienic solution.

[benkehoe]

The resource policy is part of CloudWatch Logs, so this request is essentially the same as #249. As a workaround, I think you should be able to create an IAM role with permission to deliver to CloudWatch Logs, and give that role to the target.

[ristkari]

The problem is that even if I create such a role and give it as "RoleArn" it won't get used. Only way to make it work is with resource policy on Cloudwatch.

Tried with:

  CFNTargetRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service: events.amazonaws.com
            Action: 'sts:AssumeRole'
      Policies:
      -   PolicyName: "AllowLogging"
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: "Allow"
                Action:
                  - 'logs:*'
                Resource: "*"
      -   PolicyName: "AllowLambdaInvoke"
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: "Allow"
                Action:
                  - 'lambda:InvokeFunction'
                Resource: !GetAtt EventsFunction.Arn

And the Lambda wouldn't fire either without the permission:

  EventbridgePermission:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName: !GetAtt EventsFunction.Arn
      Action: lambda:InvokeFunction
      Principal: events.amazonaws.com
      SourceArn: !GetAtt CFNRule.Arn

[benkehoe]

When you set that up, can you verify with aws events list-targets-by-rule --rule <rule-name> that the role is correctly set on the target?

[ristkari]

It doesn't look it is set on the target:

❯ aws events list-targets-by-rule --rule cfn-status-consumer-CFNRule-W6YPCCYF0LQA                                                                                           op-cloudfoundation-tools [gke_angrybeardproject_europe-north1_angrybeard|default]
{
    "Targets": [
        {
            "Id": "CloudwatchLogsTarget",
            "Arn": "arn:aws:logs:eu-central-1:REDACTEC_ACCOUNT_ID:log-group:/aws/events/CFN"
        },
        {
            "Id": "LambdaTarget",
            "Arn": "arn:aws:lambda:eu-central-1:REDACTEC_ACCOUNT_ID:function:cfn-event-lambda"
        }
    ]
}

I used the following template snippet:

  CFNRule:
    Type: AWS::Events::Rule
    Properties:
      RoleArn: !GetAtt CFNTargetRole.Arn
      EventPattern:
        source:
          - "aws.states"
      Targets:
        - Id: 'CloudwatchLogsTarget'
          Arn: !Sub "arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:${CFNLogGroup}"
        - Id: 'LambdaTarget'
          Arn: !GetAtt EventsFunction.Arn

[benkehoe]

Ah, if you put it on the role it should show up in aws events describe-rule --name <rule-name>. What happens if you put the role in the target like this?

CFNRule:
    Type: AWS::Events::Rule
    Properties:
      EventPattern:
        source:
          - "aws.states"
      Targets:
        - Id: 'CloudwatchLogsTarget'
          RoleArn: !GetAtt CFNTargetRole.Arn
          Arn: !Sub "arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:${CFNLogGroup}"
        - Id: 'LambdaTarget'
          Arn: !GetAtt EventsFunction.Arn

[ristkari]

You cannot put it there. I tried - cloudformation complains that RoleArn is not supported for Cloudwatch (nor for Lambda). However it is required if targetting eventbus on another account.

So this is why it's pretty confusing when it is possible to also provide it in properties and also on individual targets.

[benkehoe]

Wow, that's kind of a mess.

[TLaue]

I am having the same issue. Any workaround available? Thank you very much for any advice.

[alp-garcia]

Same issue here. Does anyone have an AWS CLI script as workaround?

[amirbadnjevic]

Same here. Would really prefer having the support for it in CloudFormation.
any ETA?

[shotty1]

Not sure if this is exactly the same issue, but I have played around with this for a bit and the whole thing only seems to work when the LogGroupName starts with /aws/events/
I still can not set a RoleArn for the individual target though

This definition just works in my tests:

  RuleSecurityScans:
    Type: AWS::Events::Rule
    Properties:
      Description: ""
      EventPattern:
        {
          "detail-type": [
            "ECR Image Scan"
          ],
          "source": [
            "aws.ecr"
          ]
          }
      State: ENABLED
      RoleArn: !GetAtt RoleCWLogsDelivery.Arn
      Targets:
      -
        Arn: !GetAtt LogGroupScanFindings.Arn
        Id: LogGroup


  RoleCWLogsDelivery:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service: events.amazonaws.com
            Action: 'sts:AssumeRole'
      Policies:
      -   PolicyName: "AllowLogging"
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: "Allow"
                Action:
                  - 'logs:*
                Resource: "*"

  LogGroupScanFindings:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: /aws/events/SamImageScanFindings
      RetentionInDays: 30

[rnhurt]

To add a bit of context to @shotty1's comment, you can't even use the CloudWatch Events console to send events to a "custom" log group. I have several log groups in my account and the only one the console will allow me to configure is the "TESTING" one (/aws/events/TESTING).

My CFN template tried to configure it to put events to the "/lh/securityhub/events" log group; that's where the "hub/events" snipped comes from in the text box.

tl;dr I think this might be a CloudWatch Event Rules problem and not really a CFN problem.

[ErkoKnoll]

I created L2 CDK construct for using CW as a target for EventBridge. I used a custom resource as a workaround so my Log Group would accept logs sent from EventBridge. This can be translated to CF:

export class LogGroupTarget implements IRuleTarget {
    constructor(private readonly stack: Stack, private readonly logGroup: LogGroup) {
    }
    
    public bind(rule: IRule, id: string): RuleTargetConfig {
        // Ugly hack to grant a permission for allowing EventBridge to store logs in CloudWatch
        const policyName = `${rule.ruleName}-CloudWatchPolicy`
        new AwsCustomResource(this.stack, "CloudwatchLogResourcePolicy", {
            resourceType: "Custom::CloudwatchLogResourcePolicy",
            onUpdate: {
                service: "CloudWatchLogs",
                action: "putResourcePolicy",
                parameters: {
                    policyName,
                    policyDocument: JSON.stringify({
                        Version: "2012-10-17",
                        Statement: [
                            {
                                Sid: policyName,
                                Effect: "Allow",
                                Principal: {
                                    Service: ["events.amazonaws.com"]
                                },
                                Action: ["logs:CreateLogStream", "logs:PutLogEvents"],
                                Resource: this.logGroup.logGroupArn
                            }
                        ]
                    })
                },
                physicalResourceId: PhysicalResourceId.of(policyName),
            },
            onDelete: {
                service: "CloudWatchLogs",
                action: "deleteResourcePolicy",
                parameters: {
                    policyName
                }
            },
            policy: AwsCustomResourcePolicy.fromStatements([
                new PolicyStatement({
                    actions: ["logs:PutResourcePolicy", "logs:DeleteResourcePolicy"],
                    resources: ["*"]
                })
            ])
        });

        return {
            id: '',
            arn: this.logGroup.logGroupArn,
            targetResource: this.logGroup
        }
    }
}

[josjaf]

Do we need an iam role if we are using /aws/events as the prefix?

[aditya81070]

@josjaf No, we don't need any IAM role. I just changed my log group name from my-events to /aws/events/my-events and things started to work fine.

[nickste]

Hi folks - a quick update as CloudWatch Logs now supports setting a log group resource policy via CloudFormation: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-logs-resourcepolicy.html

This CloudFormation template shows and example of setting up an EventBridge event bus, a rule, and a CloudWatch Logs target, with a resource policy that accepts events from the created EventBridge rule. There is no need to provide an IAM role on the rule.

AWSTemplateFormatVersion: '2010-09-09'
Description: Sample template to deliver events from Amazon EventBridge to CloudWatch Logs

Resources:
  TestEventBus:
    Type: AWS::Events::EventBus
    Properties:
      Name: test-bus
  LogsRule:
    Type: AWS::Events::Rule
    Properties:
      EventBusName: !Ref TestEventBus
      EventPattern:
        account:
          - !Ref AWS::AccountId
      Targets:
        - Arn: !GetAtt LogGroupForEvents.Arn
          Id: LogTarget
  LogGroupForEvents:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: /aws/events/resource-policy-test
  LogGroupForEventsPolicy:
    Type: AWS::Logs::ResourcePolicy
    Properties:
      PolicyName: EventBridgeToCWLogsPolicy
      PolicyDocument: !Sub 
      - >
        {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Sid": "EventBridgetoCWLogsPolicy",
              "Effect": "Allow",
              "Principal": {
                "Service": [
                  "delivery.logs.amazonaws.com",
                  "events.amazonaws.com"
                ]
              },
              "Action": [
                "logs:CreateLogStream",
                "logs:PutLogEvents"
              ],
              "Resource": [
                "${logArn}"
              ],
              "Condition": {
                "ArnEquals": {"aws:SourceArn": "${ruleArn}"}
              }
            }
          ]
        }
      - { logArn: !GetAtt LogGroupForEvents.Arn, ruleArn:  !GetAtt LogsRule.Arn}

[jdiamond]

Hi @nickste, I'm not able to get that policy document to work with the condition on SourceArn.

I noticed that if I add a target in the AWS console, a policy document named "TrustEventsToStoreLogEvents" gets created with a resource that ends with ":log-group:/aws/events/:" and no condition on a specific rule. So if I have that in my account, policy documents like in your example appear to work but they stop as soon as I delete that implicitly created policy.

[kpena027]

I had the same issue as @jdiamond. I tried hardcoding the resources, adding a wildcard, using Condition StringLike with aws:PrincipalArn and could not get it to work. The example on the AWS documentation doesn't include a SourceArn, so I will assume it just doesn't work currently https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-use-resource-based.html. The following policy worked:

  LogGroupForEventsPolicy:
    Type: AWS::Logs::ResourcePolicy
    Properties:
      PolicyName: EventBridgeToCWLogsPolicy
      PolicyDocument: !Sub 
      - >
        {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Sid": "EventBridgetoCWLogsPolicy",
              "Effect": "Allow",
              "Principal": {
                "Service": [
                  "delivery.logs.amazonaws.com",
                  "events.amazonaws.com"
                ]
              },
              "Action": [
                "logs:CreateLogStream",
                "logs:PutLogEvents"
              ],
              "Resource": [
                "${logArn}"
              ]
            }
          ]
        }
      - { logArn: !GetAtt LogGroup.Arn }

[piersf]

I created a rule and provided a log group as the target, without creating a log group resource policy, and it worked. The event rule was able to write to the log group just fine.

Is a log groups resource policy automatically created if one is not provided?

[jdiamond]

@piersf If you ever used the console to create a log group target, it created a resource policy for you.

[piersf]

@jdiamond im sorry, I forgot to mention that I created the above using CloudFormation.

[jdiamond]

@piersf Yeah, but I'm guessing maybe you created one in the console in the past. The first time I tried to use CloudFormation, it appeared to work because I had the resource policy created by the console but it didn't work in an account without that policy. You can check your account to see if you already have a policy that allows your rule to work.

[SaifurShatil]

just create a loggroup with the name "/aws/events/groupname" and it will work fine & also set policy logs:*

[carpusherw]

@piersf Yeah, but I'm guessing maybe you created one in the console in the past. The first time I tried to use CloudFormation, it appeared to work because I had the resource policy created by the console but it didn't work in an account without that policy. You can check your account to see if you already have a policy that allows your rule to work.

It turns out that the UI will create a resource based (or more like service based IMO) policy that contains wild card in resource name in a region. Using aws logs describe-resource-policies to list policies after creating a log group target from UI will give you this

 {
    "policyName": "TrustEventsToStoreLogEvents",
    "policyDocument": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Sid\":\"TrustEventsToStoreLogEvent\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":[\"delivery.logs.amazonaws.com\",\"events.amazonaws.com\"]},\"Action\":[\"logs:CreateLogStream\",\"logs:PutLogEvents\"],\"Resource\":\"arn:aws:logs:ap-southeast-1:<YOUR_ACCOUNT>:log-group:/aws/events/*:*\"}]}",
    "lastUpdatedTime": 1642662488316
}

Watch out the log group name, /aws/events/*:*. That's why your account then have no problems from then on in that region.