Issue in creating the Lambda invocation from S3 bucket

[anshul1790]

Issue in creating the Lambda invocation from S3 bucket

The requirement is to add a trigger in Lambda function on object creation in s3 bucket along with some VPC, s3 and cloud watch permissions, trying this using CF.

1. What did you do?
   The fresh/initial deployment works fine with all new resources build like IAM role and associated policies, able to add new s3 bucket using CF and add an event trigger/invocation in Lambda from the same s3 bucket. On next/subsequent deployments, to skip the s3 bucket creation, I added in YAML: “existing: true”, from the official documentation of serverless deployment.

2. What happened?
   It failed with below message: “Failed to create a resource. Configuration is ambiguously defined. Cannot have overlapping suffixes in two rules if the prefixes are overlapping for the same event type. See details in CloudWatch Log

3. What should've happened?
   It should be redeployed my stack as I have changed anything

4. What's the content of your serverless.yml file?

service: experiment-play-ver04

provider:
  name: aws
  runtime: python3.7
  memorySize: 1024 # optional, in MB, default is 1024
  timeout: 300 # optional, in seconds, default is 6
  versionFunctions: true # optional, default is true
  profile: ${opt:account, 'dev'} # aws account name
  region: ${opt:region, 'us-west-2'}
  stage: ${opt:stage, 'experiment'}

# Plugin to deploy python dependencies through a docker image.
# Note: Docker must be installed on your system
plugins:
  - serverless-python-requirements
custom:
  pythonRequirements:
    dockerizePip: true
  bucketRef:
    experiment: experiment-play-ver04

# Lambda functions
functions:
  ExperimentPlayVer04:
    name : experiment-play-ver04 # actual name in AWS
    handler: lambdaMain.lambda_handler
    role: ExperimentPlayLambdaRoleVer04
    events:
      - s3:
          bucket: ${self:custom.bucketRef.${self:provider.stage}}
          event: s3:ObjectCreated:*
          rules:
            - prefix: export/monitor/
            - suffix: .csv
    environment:
      BUCKET_NAME: ${self:custom.bucketRef.${self:provider.stage}}

resources:
  Resources:
    ExperimentPlayLambdaCloudwatchVer04:
      Properties:
        ManagedPolicyName: experiment-play-ver04-lambda-cloudwatch
        PolicyDocument:
          Statement:
            - Action:
                - logs:CreateLogStream
                - logs:PutLogEvents
              Effect: Allow
              Resource:
                - arn:aws:logs:*:*:log-group:*
            - Action:
                - logs:PutLogEvents
              Effect: Allow
              Resource:
                - arn:aws:logs:*:*:log-group:*:*:*
            - Action:
                - logs:CreateLogGroup
                - cloudwatch:PutMetricData
              Effect: Allow
              Resource:
                - '*'
          Version: '2012-10-17'
      Type: AWS::IAM::ManagedPolicy
    ExperimentPlayS3AccessVer04:
      Properties:
        ManagedPolicyName: experiment-play-ver04-s3-access
        PolicyDocument:
          Statement:
            - Action:
                - s3:PutAccountPublicAccessBlock
                - s3:GetAccountPublicAccessBlock
                - s3:ListAllMyBuckets
                - s3:ListJobs
                - s3:CreateJob
                - s3:HeadBucket
                - s3:GetObject
                - s3:GetObjectAcl
                - s3:PutObject
              Effect: Allow
              Resource:
                - '*'
            - Action:
                - s3:*
              Effect: Allow
              Resource:
                - { "Fn::Join" : ["", ["arn:aws:s3:::", { "Ref" : "ServerlessDeploymentBucket" }]]}
                - { "Fn::Join" : ["", ["arn:aws:s3:::${self:custom.bucketRef.${self:provider.stage}}"]]}
          Version: '2012-10-17'
      Type: AWS::IAM::ManagedPolicy
    ExperimentPlayLambdaRoleVer04:
          Type: AWS::IAM::Role
          Properties:
            Path: /service-role/
            RoleName: experimentPlayLambdaRoleVer04
            AssumeRolePolicyDocument:
              Statement:
                - Effect: Allow
                  Principal:
                    Service:
                      - lambda.amazonaws.com
                      - events.amazonaws.com
                  Action: sts:AssumeRole
            ManagedPolicyArns:
              - 'Fn::Join':
                - ':'
                -
                  - 'arn:aws:iam:'
                  - Ref: 'AWS::AccountId'
                  - 'policy/experiment-play-ver04-lambda-cloudwatch'
              - 'Fn::Join':
                - ':'
                -
                  - 'arn:aws:iam:'
                  - Ref: 'AWS::AccountId'
                  - 'policy/experiment-play-ver04-s3-access'

5. What's the output you get when you use the SLS_DEBUG=* environment variable (e.g. SLS_DEBUG=* serverless deploy)

})
CloudFormation - DELETE_IN_PROGRESS - AWS::IAM::Role - IamRoleCustomResourcesLambdaExecution
CloudFormation - DELETE_FAILED - AWS::IAM::Role - IamRoleCustomResourcesLambdaExecution
CloudFormation - UPDATE_ROLLBACK_COMPLETE - AWS::CloudFormation::Stack - experiment-play-experiment
Serverless: Operation failed!
Serverless: View the full error output: https://us-west-2.console.aws.amazon.com/cloudformation/home?region=us-west-2#/stack/detail?stackId=arn%3Aaws%3Acloudformation%3Aus-west-2%3A755529035501%3Astack%2Fexperiment-play-experiment%2F0ca3d980-74c7-11ea-9b94-0678f540705a
 
  Serverless Error ---------------------------------------
 
  ServerlessError: An error occurred: ExperimentPlayCustomS31 - Failed to create resource. Configuration is ambiguously defined. Cannot have overlapping suffixes in two rules if the prefixes are overlapping for the same event type. See details in CloudWatch Log: 2020/04/02/[$LATEST]ca5fffad2118467bbfad3fc7cc1ffa6a.
    at /usr/local/lib/node_modules/serverless/lib/plugins/aws/lib/monitorStack.js:121:33

Similar or dependent issues:

• More than one s3 event failing deployment for existing s3 bucket. #6461

[tommedema]

Any news on this? We've had the following configuration work in production, and now that I'm redeploying, the deployment fails without any changes to the serverless.yml file:

functions:
  transformWebmToMP4:
    handler: dist/transform.webmToMP4
    layers:
      - !GetAtt FfmpegLambdaLayer.Outputs.LayerVersion
    events:
      - s3:
          existing: true
          bucket: ${{env:AWS_S3_MEDIA_BUCKET_NAME}}
          # See https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html#notification-how-to-filtering
          event: s3:ObjectCreated:CompleteMultipartUpload
          rules:
            - prefix: public/
            - suffix: .webm

resources:
  Transform: AWS::Serverless-2016-10-31
  Resources:
    FfmpegLambdaLayer:
      Type: AWS::Serverless::Application
      Properties:
        Location:
          ApplicationId: arn:aws:serverlessrepo:us-east-1:145266761615:applications/ffmpeg-lambda-layer
          SemanticVersion: 1.0.0

Error we get is:

Serverless Error ---------------------------------------
 
  An error occurred: TransformWebmToMP4CustomS31 - Failed to create resource. Configuration is ambiguously defined. Cannot have overlapping suffixes in two rules if the prefixes are overlapping for the same event type. See details in CloudWatch Log: 2020/04/11/[$LATEST]07aca82713e646b4aff6b55a2c2a934e.
 
  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com
 
  Your Environment Information ---------------------------
     Operating System:          darwin
     Node Version:              12.14.1
     Framework Version:         1.67.3
     Plugin Version:            3.6.6
     SDK Version:               2.3.0
     Components Version:        2.29.1

Tried updating the serverless framework to 1.67.3 but didn't help

[medikoo]

@tommedema thanks for report. It's hard for me to tell upfront what went wrong.

It'll be great to obtain more debug information. If I read correctly error happens in custom resource lambda, and most likely crash occurs here:

serverless/lib/plugins/aws/customResources/resources/s3/lib/bucket.js

Line 77 in 8abae84

return s3.putBucketNotificationConfiguration(payload).promise();

It'll be nice to see exact payload that's rejected by AWS, having that we could probably drive a correct conclusion.

Logging of custom resource lambda can be turned on via provider.logs.frameworkLambda: true.
Having that on we may probably manually add some logs to framework code and then in CloudWatch read the outcome.

[kevlarr]

I don't know whether this is related or tangential, but we just removed a Lambda function that had an S3 event on it and attempted to deploy a different Lambda that had the same configuration. We're getting the same Ambiguous configuration error because removing the function does not remove the associated S3 event configuration.

[medikoo]

removing the function does not remove the associated S3 event configuration.

@kevlarr thanks for jumping in. Can you further clarify above part?

[kevlarr]

@medikoo Yep, sorry about not including any detail.

Assume I have two functions that respond to uploads in specific locations in the same bucket defined like...

myFunction1:
  events:
    - s3:
        bucket: my-bucket
        existing: true
        event: s3:ObjectCreated:*
        rules:
          - prefix: some/prefix/1/

myFunction2:
  events:
    - s3:
        bucket: my-bucket
        existing: true
        event: s3:ObjectCreated:*
        rules:
          - prefix: some/prefix/2/

Everything is fine; this works because the prefixes are different. If I go into S3 management console and look at that my-bucket properties, I'll see a notification registered that looks like...

name	events	filter	type
my-stack-name-myFunction1-someweirdhash	All object create events	some/prefix/1/	Lambda
my-stack-name-myFunction2-anotherhash	All object create events	some/prefix/2/	Lambda

And if I click "edit" I see the Lambda ARN pointing to each Lambda function.

Now, assume that I need to deprecate myFunction1 and replace it with another function that responds to created events in the same prefix, so my config now looks like

aDifferentFunction:
  events:
    - s3:
        bucket: my-bucket
        existing: true
        event: s3:ObjectCreated:*
        rules:
          - prefix: some/prefix/1/

myFunction2:
  events:
    - s3:
        bucket: my-bucket
        existing: true
        event: s3:ObjectCreated:*
        rules:
          - prefix: some/prefix/2/

Deploying this will fail because, while it will tear down the MyFunction1LambdaFunction resource, it will not remove the S3 bucket notification, so when it attempts to add a notification event on some-bucket/some/prefix/1/ it fails because the original notification is still listed.

Even if I attempt this across 2 deploys...

1. first deploy to tear down myFunction1
2. second deploy to add aDifferentFunction

... it again fails, so it's not related to attempting the replacement of the Lambda function within the single deploy. When I inspect the S3 bucket's notifications, I still see the old event (even after just step #⁠1 above) that points to an ARN for a Lambda that no longer exists.

I can delete the notification manually and then deploy #⁠2 above and it technically works, but it should be automated for (1) consistency, and (2) because we only have console access to 1 of 3 AWS accounts for a client, and now we need to instruct them to manually delete the notification, and getting that step wrong will break our application (the notifications are business-critical.)

[kevlarr]

Okay, I think you can ignore me. I'm watching the verbose CF output right now and I'm seeing something similar to the following (with names changed)..

CloudFormation - DELETE_IN_PROGRESS - AWS::CloudFormation::CustomResource - MyFunction1CustomS31
CloudFormation - DELETE_FAILED - AWS::CloudFormation::CustomResource - MyFunction1CustomS31
CloudFormation - DELETE_IN_PROGRESS - AWS::CloudFormation::CustomResource - MyFunction1CustomS31
CloudFormation - DELETE_FAILED - AWS::CloudFormation::CustomResource - MyFunction1CustomS31

This is buried in the middle of other updates and it isn't causing the stack update to fail, so I simply never saw that Serverless had attempted (twice, actually) to remove the S3 event for the deleted Lambda.

[tommedema]

This is still an issue.

We just had it again and I've created a screen recording with details here:

https://app.usebubbles.com/7Gio7XbzsTteyHEXcM848M/comments-on-us-west-2-console-aws-amazon-com

Basically, when the serverless framework creates a custom resource to handle the "existing" s3 bucket triggers, a next deployment may fail when the event rules have changed due to a false "overlapping events" error. In our case we removed all event triggers except one, and still had the issue (all described in above screen recording).

[anshul1790]

Thanks Tom, indeed it helped and it worked fine for me.

On Mon, 6 Jul 2020 at 11:20 PM, Tom Medema ***@***.***> wrote: This is still an issue. We just had it again and I've created a screen recording with details here: https://app.usebubbles.com/7Gio7XbzsTteyHEXcM848M/comments-on-us-west-2-console-aws-amazon-com Basically, when the serverless framework creates a custom resource to handle the "existing" s3 bucket triggers, a next deployment may fail when the event rules have changed due to a false "overlapping events" error. In our case we removed all event triggers except one, and still had the issue (all described in above screen recording). — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#7536 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKY3HLFCEOOGK7CF5T5IGTTR2IFFTANCNFSM4L2W43SQ> .

-- Anshul Choudhary

[maxgr0]

Any news on this? Facing the same bug.
Some further investigation points to a problem with the associated IAM-Role (not enough permissions to delete parts of the resource)
@anshul1790 @tommedema @kevlarr @medikoo

[medikoo]

@maxgr0 can you prepare some simple reproduction case? Ideally through a simple service with no plugins involved.

[cscetbon]

Same here @medikoo, using serverless-plugin-existing-s3 I don't have any issue but switching to the native way doesn't work ...

An error occurred: SyncCustomS31 - Failed to create resource. Configuration is ambiguously defined. 
Cannot have overlapping suffixes in two rules if the prefixes are overlapping for the same event type

[medikoo]

@cscetbon can you prepare a reproduction case?

[isgallagher]

I just solved my issue by deleting the event notifications on the bucket that applied to my functions.

[ps2goat]

A simple repro is:

1. create a bucket (manually, outside of serverless)
2. create a function (this and the rest are using serverless)
3. deploy it. sls deploy
4. add the configuration for an s3 trigger to an existing bucket for this function.
5. deploy just the function (nothing happens) sls deploy function -f my-function
6. deploy just the function config (nothing happens) sls deploy function -f my-function -u
7. deploy the entire app (nothing happens) sls deploy
8. destroy the entire app sls remove
9. redeploy the app sls deploy, now the trigger is created.

I haven't dug into it a whole lot, but it seems like it only works when there's a new full stack created.

Updated with sample setup.

We are using serverless version 2.66.1.

We do have the following plugins being used:

• serverless-esbuild
• serverless-plugin-log-subscription

Here's a sample of what I'm running, this being just the functions part.

    # just to show I have two functions
    my-first-function:
    #... parameters omitted
    my-function:
        handler: src/handlers/my-function.handler
        timeout: 60
        events:
            - s3:
                bucket: my-existing-bucket
                event: s3:ObjectCreated:*
                existing: true

Sample issue 2:

After a successful deployment above (with the s3 trigger), commenting out the events node and redeploying the function does not change anything.

    # just to show I have two functions
    my-first-function:
    #... parameters omitted
    my-function:
        handler: src/handlers/my-function.handler
        timeout: 60
        # events:
        #     - s3:
        #         bucket: my-existing-bucket
        #         event: s3:ObjectCreated:*
        #         existing: true

In fact, the CLI outputs this message:

Serverless: Configuration did not change. Skipping function configuration update.

If I redeploy the entire app (sls deploy), then the config change is found and the trigger is removed.

[pgrzesik]

Hello @anshul1790 👋 The deploy function command is limited when it comes to functionality and it does not handle removal/creation of events.