How to reference an existing s3 bucket & prefix #124
Comments
|
Hi, At this moment SAM does not support an existing bucket as an event source. It is mentioned in the documentation: "NOTE: To specify an S3 bucket as an event source for a Lambda function, both resources have to be declared in the same template. AWS SAM does not support specifying an existing bucket as an event source." |
|
Hmm ok, but I do need to have the lambda listen on an existing bucket/folder, what are the walkarounds if I still want to make use of SAM? |
|
I just came across this too. Will this be fixed? This makes SAM very hard to use for the S3 event use case. |
|
I want to up-vote this well. At the moment this is a showstopper for me too. |
|
This is not supported in Cloudformation. Basically, cloudformation cannot change any aws resource outside of the stack. Unfortunately, as of now, there is no workaround for this limitation. |
|
@vikrambhatt do you think AWS will come out with any tooling on top of SAM/CFN to assist with cases such as this. It does make SAM hard to use unfortunately. |
|
It is possible to have CFT send parameters to and invoke a lambda process where you can programmatically make changes to existing resources. However, note that your lambda has to also be able to delete any changes it makes when the stack is deleted (CF will not know about these changes and cannot auto-delete them). |
|
Yeah, CloudFormation folks are aware of this limitation and working to solve it. We don't have an ETA yet, but I want to let you guys know that this is in the works. I am going to close this Issue because SAM is helpless without the CFN feature. |
|
I know this issue is closed, but, when will we have some news about it ? |
|
Faced this limitation recently and workaround it with combination of S3->SNS and SAM<-SNS. Works pretty good and completely automated. |
|
Thanks @sanyer for your feedback !
It's a workauround too, but I hope it helps somebody. |
|
@rzijp I'll try to remember and find where and how it was done. |
|
For existing s3 bucket, you can use this serverless plugin. |
|
Well this just sucks. We should be able to specify arn references to existing buckets. You allow it for ManagedPolicyArns for IAM ROLE, referencing a bucket shouldn't be an issue...unless a change is being made on the bucket itself? |
|
If that's the case then the documentation is incorrect, since it gives an example of referencing a bucket that is not managed by SAM. |
it has clearly said "to specify an S3 bucket as an event source for a Lambda function, both resources have to be declared in the same template. AWS SAM does not support specifying an existing bucket as an event source." We need the feature to reference an existing s3 bucket |
|
The limitation of CloudFormation makes complete sense to me. I was initially upset to hit this limitation myself and put my head down on what I think is a good workaround that fits both CloudFormation and SAM best practices. Of course, using Bash as a little bit of IaC glue as needed. The solution, first use no event in your template.yaml file and also add permission for the S3 bucket to invoke the function. To also make this work, output the functions arn. Pretty much what @helenoalves shared. Resources:
MyFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: .
Handler: app.handler
Runtime: ruby2.5
ImageBucketPermission:
Type: AWS::Lambda::Permission
Properties:
Action: 'lambda:InvokeFunction'
FunctionName: !Ref MyFunction
Principal: s3.amazonaws.com
SourceAccount: !Ref 'AWS::AccountId'
SourceArn: !Sub arn:aws:s3:::my-bucket-name
Outputs:
MyFunctionArn:
Description: My Function Arn
Value: !GetAtt MyFunction.ArnIn order to connect the S3 events a little one time Bash script, I usually put these in the projects FUNCARN=$(aws cloudformation describe-stacks \
--stack-name "my-stack-name" \
--query 'Stacks[0].Outputs[0].OutputValue'
)
JSON=$(cat <<-EOF
{
"LambdaFunctionConfigurations": [
{
"Id": "MyEventsName",
"LambdaFunctionArn": ${FUNCARN},
"Events": [
"s3:ObjectCreated:*"
]
}
]
}
EOF
)
aws s3api \
put-bucket-notification-configuration \
--bucket="my-bucket-name" \
--notification-configuration "$JSON" |
|
Just another "me too". I hit this today. |
|
I hit this issue recently and used the solution proposed above by @metaskills as a workaround. I've written a Bash script to make the whole thing a bit simpler. Hopefully someone else that lands on this thread will find it useful, and if anyone wants to suggest an improvement then please do: https://gist.github.com/tomcant/c31a08123673e91d9560737f4380cff0. Here's the script usage information: The script uses the AWS CLI (tested with version 1.16.276) so you'll need to supply valid AWS credentials for the account containing the resources. How you invoke the script depends on how you supply your credentials. I usually set the or... If you provide your credentials in some other way (e.g. EC2 instance metadata) then running the script without the extra environment variables should work just fine. The script also takes care of adding permissions for S3 to invoke the function, if necessary. The biggest limitation right now is that the script doesn't support setting filters on the notification (e.g. path prefix/suffix), but that can easily be updated on the bucket UI afterwards. Note that the jq JSON processor is also required. |
|
Ran into this today too, kinda disappointed this isn't supported. |
|
+1 |
|
Need this functionality for one of our use cases, hope this will be considered soon. |
|
+1 |
1 similar comment
|
+1 |
|
@djm I looked into that for this particular problem. It works if you only have one stack that needs to reference the bucket; if you want to reference it from multiple stacks, however, it doesn't work and another solution is needed. |
|
Can't believe this is still an issue. It could have simply been solved by adding an extra property to each resource called: |
|
+1 |
|
+1 @CJohnsonLehi : |
|
+1. Just to note that this functionality/issue was requested 2 and a half years ago. |
|
+1 |
|
honestly i'm about to abandon SAM and just go straight cloud formation. without this feature SAM is useless |
|
For anyone looking for a workaround: I ran my CloudFormation template without the event, and added the event afterward manually on the aws console. It's not ideal, but I only had to do it once, and I have been able to update the code in the lambda function without issues. |
|
+1 Still an issue. |
|
+1 |
|
+1 |
|
+1 Still an issue. |
|
...Still an issue. |
|
+1 |
3 similar comments
|
+1 |
|
+1 |
|
+1 |
|
going for a new bucket rather than waiting |
|
Check this example: S3-to-EventBridge |
|
Much needed feature !! |
|
What a disappointment... switching to serverless now, sry guys. |
|
+1 |
1 similar comment
|
+1 |
|
+1 :( |
|
+1 |
1 similar comment
|
+1 |
|
Since there is still activity here: Please leave a +1 (not leave a comment but the reaction on the top comment in the issue) on this issue: aws-cloudformation/cloudformation-coverage-roadmap#79, as SAM cannot do anything due to the lack of support in CloudFormation. |
and others
From this example: https://github.com/awslabs/serverless-application-model/blob/master/examples/2016-10-31/s3_processor/template.yaml, it creates a new bucket. However I need to reference an existing bucket, for example, I want to trigger the lambda when a *.yaml file is uploaded to
s3:/mybucket/folder?The text was updated successfully, but these errors were encountered: