AWS::S3::BucketNotification

[benkehoe]

1. Title -> AWS::S3::BucketNotification
2. Scope of request -> Allow bucket notifications to be managed separate from the bucket resource itself, resolving a longstanding circular reference problem
3. Expected behavior -> I should be able to create auto-named buckets with notifications that invoke Lambda/SNS/SQS
5. Links to existing API doc -> see below
6. Category tag -> Compute, Storage
7. Additional context:

The problem:

1. Image thumbnailing is serverless 101. It involves setting up bucket notifications to invoke a Lambda function on file upload to a bucket (then generate the thumbnails and write them back to the bucket).
2. A best practice for CloudFormation is to let CloudFormation name your resources wherever possible, and only deal with logical ids, not physical resource ids.
3. These two things cannot currently be accomplished simultaneously. There needs to be a Lambda permission or SNS/SQS topic/queue policy, which needs to reference the bucket name, but the permission is checked for at notification configuration creation, before the bucket name could be provided to the permission resource.

Fundamentally, this is because there is not a separation between the the creation of a bucket (and its name) and the settings on that bucket. There are at least three separate places on AWS that say 🤷 to customers and tell them to manually create a bucket name in two separate places, which is brittle both in terms of multiple deployments of the template and in terms of updating that bucket name in the future.

This could instead be solved with a separate BucketNotification resource. The bucket resource would be created first, the name !Ref'd to the relevant places, and then the BucketNotification resource would install the notification configuration onto the bucket.

[rosskarchner]

Would this help solve this SAM issue? aws/serverless-application-model#138

[benkehoe]

Yes, it's exactly that same problem.

[rosskarchner]

AWS folks, any chance this will move onto the board soon?

[luiseduardocolon]

We're keeping an eye on the +1s on this, but we're trying to prioritize coverage items first.

[tebruno99]

This took us by surprise today. Seems to make Policy Templates unusable.. Please Fix! We don't like letting workaround hacks live in our production environments.

[dennisandersen]

+1 on this. This has hit us more than once and feel this should be prioritized. It is not possible to achieve what i consider "THE" base use-case for bucket notifications: "read file that was just added to bucket", without resolving to cumbersome workarounds.

[TonyFNZ]

A couple of useful links:

Existing Custom Resource which implements this functionality:
https://aws.amazon.com/premiumsupport/knowledge-center/cloudformation-s3-notification-lambda/

CDK issue which is blocked by this issue: aws/aws-cdk#4323

[purnesh]

+1

[major-fire]

+1

[jorgeandresvasquez]

+1

[yeDor]

+1

[jorgeandresvasquez]

+1

[jeffmarcinko]

+1

Still implementing workarounds like this, https://aws.amazon.com/blogs/mt/resolving-circular-dependency-in-provisioning-of-amazon-s3-buckets-with-aws-lambda-event-notifications/

Plus-one for all of Ben's original points.

[IanShoe]

+1

[adhandharia]

+1

[sahil-gt]

+1

[kz974]

Yes! +1

[jamescarignan]

+1

[benbridts]

@purnesh @yeDor @IanShoe @kz974 @jamescarignan

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request

[pam81]

+1

[fwanghe]

+1

[anowac01]

How was this issue resolved? I don't see any updates in the CloudFormation documentation relevant to it, and it still warns against the circular dependency:
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-s3-bucket-notificationconfig.html

[zwezheng]

+1

[gnobre]

+1

[Us3rname]

+1

[qcurtemanjc]

+1

[agniswarmandal]

+1

[benkehoe]

@zwezheng @gnobre @Us3rname @qcurtemanjc @agniswarmandal (and future potential "+1" commenters)

If you react with the 👍 button to the original issue, (the first comment, click on the smiley face if you're the first reacting), your votes can be used to sort issues and determine priorities.

A comment will send a notification to everyone (participants and watchers), but cannot be easily counted as a vote for an issue. Thus It's generally better to vote than to comment with "+1". To keep up to date, you can also add yourself as a watcher.

[polaskj]

It looks like a more elegant solution for this is finally here with this announcement, using EventBridge. https://aws.amazon.com/blogs/aws/new-use-amazon-s3-event-notifications-with-amazon-eventbridge/

Here's a pseudo Lambda invocation example I tested with success. This assumes your externally referenced S3 Bucket enables EventBridge

Stack:

EventRule:
  Type: AWS::Events::Rule
  Properties:
    Description: EventRule
    State: ENABLED
    EventPattern: # https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-patterns.html#eb-filtering-data-
      source:
        - aws.s3
      detail-type:
        - "Object Created"
      detail:
        bucket:
          name:
            - "EXTERNAL-BUCKET"
    Targets:
      - Arn: !GetAtt MyLambda.Arn
        Id: MyLambdaFunctionTarget

PermissionForEventsToInvokeLambda:
  Type: AWS::Lambda::Permission
  Properties:
    FunctionName: !Ref MyLambda
    Action: lambda:InvokeFunction
    Principal: events.amazonaws.com
    SourceArn: !GetAtt EventRule.Arn

MyLambda:
  Type: AWS::Serverless::Function
  ... 

Bucket Stack:

S3Bucket:
  Type: AWS::S3::Bucket
  Properties:
    ...
    BucketName: "EXTERNAL-BUCKET"
    NotificationConfiguration:
      EventBridgeConfiguration:
        EventBridgeEnabled: true

Linking back to aws/serverless-application-model#124 since this is almost exactly the use-case.

[souvikataws]

Thanks for the feedback. We have created a Product Feature Request which will be prioritized with other features planned for Amazon S3. As another option, you can enable EventBridge notifications on the S3 Bucket (there are details on getting started here). Additionally, you can refer to an example from Polaskj who has provided a CloudFormation template using EventBridge and Lambda as a destination.

[XrayBravoGolf]

The related (downstream) SAM issue aws/serverless-application-model#124 has been around since 2017 and #79 has been on the Coming Soon ™️ project board for a while.
Any updates? Timelines?

I see many AWS members at this thread then have moved on, so I'd appreciate anybody who is able to check in on the status of the feature request / issue. @benkehoe is this something you are able to help?