bug: S3 bucket.add_event_notification() does not deploy

[robertchinezon]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

(CDK Local)
I opened a ticket here, but there has been no investigation.
Issue initially depicted here: localstack/aws-cdk-local#88

As the title suggests, calling add_event_notification() with any target results in the following error as seen towards the end of the log file:

2023-09-28T22:41:21.978  WARN --- [uncthread128] l.s.c.deployment_utils :  No resource provider found for "Custom::S3BucketNotifications". 

localstack.log

Expected Behavior

If I set the notification through awslocal cli:

awslocal s3api put-bucket-notification-configuration --bucket bucket --notification-configuration file://notification.json

all my code works properly.

How are you starting LocalStack?

With a docker-compose file

Steps To Reproduce

Code

"""AWS stack that sends an sqs notification when files land in a s3 bucket."""

import logging
import os

# from utils.utils import enable_my_environment, progress_bar
from aws_cdk import (
    CfnOutput,
    Duration,
    RemovalPolicy,
    Stack,
    aws_iam as iam,
    aws_lambda as _lambda,
    aws_s3 as s3,
    aws_s3_notifications as s3n,
    aws_sqs as sqs,
)
import constructs as core

logger = logging.getLogger(__name__)

# enable_my_environment()

class InputStack(Stack):
    """Instantiate an AWS input stack."""

    def __init__(self, scope: core.Construct, id: str, **kwargs) -> None:
        """Initialize class attributes."""

        super().__init__(scope, id, **kwargs)
        curr_dir = os.path.dirname(os.path.abspath(__file__))

        # Create the S3 bucket
        bucket = s3.Bucket(
            self,
            "InputBucket",
            removal_policy=RemovalPolicy.DESTROY,
            auto_delete_objects=False,
        )

        # Create the SQS queue
        queue = sqs.Queue(
            self,
            "ProcessingQueue",
            visibility_timeout=Duration.seconds(300),
        )

        # Add an IAM role for the Lambda to talk to the S3 bucket.
        lambda_role = iam.Role(
            self, "LambdaRole",
            assumed_by=iam.ServicePrincipal("lambda.amazonaws.com"),
            description="Role for Lambda function to upload files to S3 bucket"
        )

        # Add permissions to the role
        lambda_role.add_to_policy(
            iam.PolicyStatement(
                effect=iam.Effect.ALLOW,
                actions=[
                    "s3:*",
                ],
                resources=[
                    f"{bucket.bucket_arn}/*",
                    f"{bucket.bucket_arn}"
                ]
            )
        )

        # Create the Lambda function
        function = _lambda.Function(
            self,
            "MyFunction",
            runtime=_lambda.Runtime.PYTHON_3_8,
            handler="handler.lambda_handler",
            code=_lambda.Code.from_asset(os.path.join(curr_dir, "lambda/sqs_notifier")),
            environment={
                "QUEUE_URL": queue.queue_url,
            },
            role=lambda_role,
        )


        # Attach the policy to the bucket
        bucket.add_to_resource_policy(
            iam.PolicyStatement(
                effect=iam.Effect.ALLOW,
                principals=[iam.ArnPrincipal(lambda_role.role_arn)],
                actions=[
                    "s3:*",
                ],
                resources=[
                    f"{bucket.bucket_arn}/*",
                    f"{bucket.bucket_arn}"
                ]
            )
        )

        queue.grant_consume_messages(iam.ArnPrincipal(lambda_role.role_arn))

        # Add the S3 event notification to the bucket
        bucket.add_event_notification( ##################Problem Function
            s3.EventType.OBJECT_CREATED,
            s3n.SqsDestination(queue),
            s3.NotificationKeyFilter(prefix="flight", suffix=".csv"),
        )

        # Add the Lambda source_mapping
        function.add_event_source_mapping(
            id="SqsNotifier",
            event_source_arn=queue.queue_arn,
        )

        # Output the bucket name and queue URL
        CfnOutput(self, "BucketName", value=bucket.bucket_name)
        CfnOutput(self, "QueueUrl", value=queue.queue_url)
        CfnOutput(self, "QueueArn", value=queue.queue_arn)

Environment

- OS: Ubuntu 22.04
- LocalStack: 2.2.0
CDK-2.97.0
CDK-Python
Python 3.10.12

Anything else?

No response

[localstack-bot]

Welcome to LocalStack! Thanks for reporting your first issue and our team will be working towards fixing the issue for you or reach out for more background information. We recommend joining our Slack Community for real-time help and drop a message to LocalStack Pro Support if you are a Pro user! If you are willing to contribute towards fixing this issue, please have a look at our contributing guidelines and our contributing guide.

[kazazor]

I believe I experienced the same issue. I shared many new helpful details about it in the Slack channel: https://localstack-community.slack.com/archives/CMAFN2KSP/p1701200973470049

Any progress was made on this topic?

We would like to purchase Pro licenses for the company but when trying to run our CDK it breaks and is currently a blocker for the purchase.

[roeysha]

Same here! Please fix ASAP, this is critical functionality.

[laurence-myers]

CDK adds a "custom resource" to handle the notification "glue" (so to speak). Custom resources are only available in LocalStack Pro, no the free version.

Here's a hack to bypass the custom resource created by CDK. We can access the underlying L1 CDK construct, and from there set notificationConfiguration.lambdaConfigurations.

This example is in TypeScript.

    const bucket = new s3.Bucket(...);
    const lambda = new NodejsFunction(...);
    if (isDeployingToLocalStack()) { // implement or replace `isDeployingToLocalStack()` as required
      const cfnBucket = bucket.node.defaultChild as s3.CfnBucket;
      cfnBucket.notificationConfiguration = {
        lambdaConfigurations: [
          {
            creationStack: [],
            event: 's3:ObjectCreated:*',
            function: lambda.functionArn,
          },
        ],
      };
    }

This works for newly created S3 and Lambda resources.

I don't know what impact it'll have if you bypass the inline Python script added by CDK. (In the worst case scenario, you can add your own.)

[bentsku]

Hello and thanks for the report.

As it has been answered in the linked issue and by @laurence-myers who provided a nice workaround, CDK will create a Custom Resource which is Pro only.

By using the workaround, you should be able to get it working with community.

I'll close the issue as this has been answered. Thanks a lot!