Reducing Orphaned LogGroups when destroying a Stack

[ciaransweet]

TL;DR:

Do you constantly find yourself running cdk destroy and finding that you're left with 100s of LogGroups orphaned? Fed up of polluting your CloudWatch with the LogGroups of Lambdas long dead? I may have the solution for you!

Create your LogGroups named as they would be by CloudWatch! This will override the out-of-the-box ones and you can set retention & removal_policy!

An Example

I've found in the past that creating and destroying multiple stacks in a day, for multiple days, for multiple weeks results in 100s if not 1000s of orphaned LogGroups. This is annoying as it's noisy and you spend longer looking for the one LogGroup you need.

After some discussions with @JakeHendy, it was noted that creating a LogGroup explicitly, named the same as it would be implicitly, would allow us to:

• Set a custom retention policy for logs
• Set a custom removal policy for LogGroups

You can see an example below:

from aws_cdk import (
    aws_lambda,
    aws_lambda_python,
    aws_logs,
    core,
)

lambda_function = aws_lambda_python.PythonFunction(
    self,
    entry="function_entry",
    handler="handler",
    runtime=aws_lambda.Runtime.PYTHON_3_8,
    memory_size=128,
    timeout=core.Duration.minutes(5),
)

aws_logs.LogGroup(
    self,
    log_group_name=f"/aws/lambda/{lambda_function.function_name}",
    removal_policy=core.RemovalPolicy.DESTROY,
    retention=aws_logs.RetentionDays.ONE_DAY,
)

This will then result in you being able to do cdk deploy and cdk destroy and no LogGroups being available under /aws/lambda/your_function_name* 🎉

A Gotcha

One issue I am still currently struggling with (and am losing hope on) is finding an easy way to not get orphaned LogGroups for CustomResource Lambda functions.

I have implemented the same pattern for explicit LogGroup creation for a Lambda-backed CustomResource and I always end up with an orphaned LogGroup... taunting me. 🤨 👎

I believe that this is due to how CloudFormation handles the deletion of the CustomResource, it probably looks like:

• cdk destroy
• Explicit LogGroup destroyed
• CustomResource invoked with destroy
  • Lambda invoked, which creates a new (out of our domain) LogGroup
• CustomResource destroyed
• Lambda destroyed
• cdk destroy ends
• Orphaned LogGroup taunts me 🧟‍♂️

I've posted in https://cdk.dev/ and as of yet, have no real luck. I've tried bouncing around with dependencies similar to how @alukach is doing it in the ss-orders project (Making a Construct explicitly depend on another) but that's not been too successful 😭

Jake suggested creating a Lambda that listens for DeleteStack in CloudTrail then deletes all LogGroups, but that's a Lambda for the sake of handling potentially one orphaned LogGroup, so in terms of effort vs. return, it's not a winner.

Happy for folks to jump in and give suggestions on how we could cover this gotcha off!

[JakeHendy]

CloudTrail as an event source is quite powerful, we trigger a lambda from the CreateLogGroup event to ensure a log group has the right log subscriptions to forward to our central logging/auditing platform. If you need some code for it let me know and I'll dig out something I wrote :)

[ciaransweet]

@JakeHendy I'd definitely appreciate anything that could help kick off the DeleteStack idea, if that's a nicer solution, it'd cut down the LogGroups we create explicitly and we could add it as a generic construct within cdk-seed 🎉

[JakeHendy]

  LogGroupCreationEvent:
    Type: AWS::Events::Rule
    Properties:
      Name: LogGroupCreationEvent
      Description: Triggers when the CreateLogGroup API is called
      EventPattern:
        source:
          - "aws.logs"
        detail-type:
          - "AWS API Call via CloudTrail"
        detail:
          eventSource:
            - "logs.amazonaws.com"
          eventName:
            - CreateLogGroup
      State: ENABLED
      Targets:
        - Arn: !GetAtt ALambda.Arn
          Id: aLambdaArn

Can only find it in yucky yaml as opposed to CDK, but it's quite easy to port. We set the Target ID to A Lambda Arn because it was easy and we didn't need to inspect it in the console. Let me know if you want a CDK port of it and I can quickly whip one up

[ciaransweet]

I guess it's gonna use an event source like cloudformation.amazonaws.com and an event name of DeleteStack right?

[JakeHendy]

@ciaranevans yeah pretty much. https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-api-logging-cloudtrail.html