How to reference an existing s3 bucket & prefix

[wliao008]

From this example: https://github.com/awslabs/serverless-application-model/blob/master/examples/2016-10-31/s3_processor/template.yaml, it creates a new bucket. However I need to reference an existing bucket, for example, I want to trigger the lambda when a *.yaml file is uploaded to s3:/mybucket/folder?

[vikrambhatt]

Hi,

At this moment SAM does not support an existing bucket as an event source. It is mentioned in the documentation:
https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#s3

"NOTE: To specify an S3 bucket as an event source for a Lambda function, both resources have to be declared in the same template. AWS SAM does not support specifying an existing bucket as an event source."

[wliao008]

Hmm ok, but I do need to have the lambda listen on an existing bucket/folder, what are the walkarounds if I still want to make use of SAM?

[iconara]

I just came across this too. Will this be fixed? This makes SAM very hard to use for the S3 event use case.

[sbarski]

I want to up-vote this well. At the moment this is a showstopper for me too.

[vikrambhatt]

This is not supported in Cloudformation. Basically, cloudformation cannot change any aws resource outside of the stack. Unfortunately, as of now, there is no workaround for this limitation.

[sbarski]

@vikrambhatt do you think AWS will come out with any tooling on top of SAM/CFN to assist with cases such as this. It does make SAM hard to use unfortunately.

[mwarner1]

It is possible to have CFT send parameters to and invoke a lambda process where you can programmatically make changes to existing resources. However, note that your lambda has to also be able to delete any changes it makes when the stack is deleted (CF will not know about these changes and cannot auto-delete them).

[sanathkr]

Yeah, CloudFormation folks are aware of this limitation and working to solve it. We don't have an ETA yet, but I want to let you guys know that this is in the works.

I am going to close this Issue because SAM is helpless without the CFN feature.

[helenoalves]

I know this issue is closed, but, when will we have some news about it ?

[sanyer]

Faced this limitation recently and workaround it with combination of S3->SNS and SAM<-SNS. Works pretty good and completely automated.

[helenoalves]

Thanks @sanyer for your feedback !
I never used SNS before, amazon has a lot of amazing tools and I dont know all.
Me and my friend @alexiscviurb , an amazing infrastructure engineer, created a script to automate the task in a Jenkins Pipeline.
This shell script is doing this steps:

1. Create With Cloud Formation The Functions

2. List stack resources:
   aws cloudformation list-stack-resources --stack-name analytics-functions --query 'StackResourceSummaries[?LogicalResourceId=='$1'].[PhysicalResourceId]' --output text

3. Get Lambda Functions:
   aws lambda list-functions --query 'Functions[?FunctionName=='$FunctionName'].[FunctionArn]' --output text

4. Replace in SAM JSON FunctionName
   sed -i "s/FunctionName::ARN/$FunctionName/" sam-configuration.json

5. Remove old Functions of Bucket
   aws s3api put-bucket-notification-configuration --bucket=bucket-name --notification-configuration="{}"

6. Bind the new Functions with Bucket
   aws s3api put-bucket-notification-configuration --bucket=bucket-name --notification configuration file://sam-configuration.json

It's a workauround too, but I hope it helps somebody.
Regards,
Heleno

[rzijp]

@sanyer, any chance that you could share more details about your automation? Did you use SAM (not sure if this is limited by #249 as well), or other means?