New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't subscribe to events of existing S3 bucket #2154

Closed
kennu opened this Issue Sep 16, 2016 · 34 comments

Comments

Projects
None yet
@kennu
Contributor

kennu commented Sep 16, 2016

This is a Bug Report

Description

When specifying an already existing S3 bucket (xxx) in a function event, I get this error on deploy:

- xxx already exists.

I could not find documentation on how to subscribe to events of an existing S3 bucket in 1.0 rc1.

This worked in 0.5.

@flomotlik

This comment has been minimized.

Contributor

flomotlik commented Sep 20, 2016

Its not possible to do this through Cloudformation. Basically you're not able to change existing infrastructure through CF.

On a more fundamental level I think its also best practice to create/change configuration of a resource only in one place. While we will probably have the functionality to add events to existing infrastructure I don't think its a good thing to do in general, as then you create very strong dependencies and expectations between different services, e.g. if you rename a piece of infrastructure or you want to rename the S3 bucket you would have to update every serverless service that references the bucket.

I'm not sure that it will be a good experience for users.

I'm going to close here as this functionality is not going to be resolved soon, but would be interested in getting your thought on how to integrate with existing infrastructure through Serverless.

@flomotlik flomotlik closed this Sep 20, 2016

@kennu

This comment has been minimized.

Contributor

kennu commented Sep 20, 2016

:-( This has been a frequent use case in my projects on 0.5. There is quite often existing infrastructure that I want to integrate to a Serverless project. Sorry to hear this will no longer be possible.

@kennu

This comment has been minimized.

Contributor

kennu commented Sep 20, 2016

Overall, it basically sounds like quite many things that 0.5 was an excellent tool for will no longer be possible in 1.0. It makes me wonder if they will be doable with plugins. I don't really like the old days of writing shell scripts, which Serverless 0.x pretty much eliminated.

@andymac4182

This comment has been minimized.

Contributor

andymac4182 commented Sep 20, 2016

There is a plugin for handling this https://github.com/matt-filion/serverless-external-s3-event It should do what you want @kennu

@flomotlik

This comment has been minimized.

Contributor

flomotlik commented Sep 20, 2016

As @andymac4182 said definitely possible. (btw didn't know this one existed, good find)

I'm basically just VERY hesitant to introduce features into Serverless that bind a stack to existing infrastructure, because it will very quickly lead to issues with other infrastructure being renamed, removed and how Serverless then tries to sync this.

Now of course to introduce Serverless into an existing infrastructure this has to be possible to some degree, so we'll have some of those features (or introduce community plugins that do this), but I'm not sure if it makes sense to put it in the core.

@kennu

This comment has been minimized.

Contributor

kennu commented Sep 20, 2016

@andymac4182 Thanks, I will look at it.

@flomotlik I've experienced the joys of updating CloudFormation stacks and seeing them enter invalid states or accidentally deleting resources, and it's the reason why I keep advocating against using it for too complex things in the first place. But this discussion has already been had earlier. :-)

@bohnman

This comment has been minimized.

bohnman commented Oct 12, 2016

@flomotlik I get what you are saying about dependency binding, but as @kennu mentioned, I think it's unrealistic to require users to use Cloud Formation in order to subscribe to s3/sns/etc. events. I work for a company that has 5 developers, of which I am one. At this point, we don't have time to learn and write Cloud Formation templates.

The plugin that @andymac4182 is very buggy.

@flomotlik

This comment has been minimized.

Contributor

flomotlik commented Oct 12, 2016

@bohnman Totally get that this isn't an option or worthwhile investment for everyone, but we have to focus on where we can make the most difference and implementing Cloudformation has allowed us to move much faster than without it and many small, medium and very large companies are using it. This allows us as a project but also as a business to grow into those companies because there is some standard that we can easily follow and build upon.

If we had enough resources to do all of it I'd like to, but we have to focus and integrating with the AWS API and managing the state of resources ourselves is simply something that is a lot of work with very little upside for us and many (probably most) of our users, so we can't do it for the most part at the moment.

@jgardezi

This comment has been minimized.

jgardezi commented Nov 25, 2016

I am having the same issue. Unable to subscribe to existing S3 bucket.

I have check this plugin https://github.com/matt-filion/serverless-external-s3-event and it is very buggy.

Is their an alternative solution to this.

Warm regards,
Javed Gardezi

@pmuens

This comment has been minimized.

Member

pmuens commented Nov 25, 2016

@jgardezi thanks for replying. 👍
Could you share your serverless.yml file so that we can take a look?

Thanks in advance!

@jgardezi

This comment has been minimized.

jgardezi commented Nov 25, 2016

Thank for the quick reply

When I used this plugin https://github.com/matt-filion/serverless-external-s3-event my serverless.yml looked like this

service: uploaded

frameworkVersion: ">=1.1.0 <2.0.0"

plugins:
  - serverless-external-s3-event

provider:
  name: aws
  runtime: nodejs4.3

  stage: dev
  region: ap-southeast-2

  iamRoleStatements:
    - Effect: "Allow"
      Action:
        - "s3:ListBucket"
        - "s3:PutObject"
      Resource: { "Fn::Join" : ["", ["arn:aws:s3:::", { "Ref" : "ServerlessDeploymentBucket" } ] ]  }
    - Effect: "Allow"
      Action:
        - "s3:PutBucketNotification"
      Resource:
        Fn::Join:
          - ""
          - - "arn:aws:s3:::mybucket"

functions:
  pgsave:
    handler: postgresql.pgsave
    timeout: 60
    events:
    - existingS3:
        bucket: mybucket
        bucketEvents:
          - s3:ObjectCreated:*

Without the above plugin my serverless.yml looked like below.

service: uploaded

frameworkVersion: ">=1.1.0 <2.0.0"

provider:
  name: aws
  runtime: nodejs4.3

  stage: dev
  region: ap-southeast-2

functions:
  pgsave:
    handler: postgresql.pgsave
    events:
    - s3:
        bucket: mybucket
        event: s3:ObjectCreated:*

The error I was getting without the plugin yml file

.........Serverless: Deployment failed!
 
  Serverless Error ---------------------------------------
 
     An error occurred while provisioning your stack: S3BucketDevcollectory
     - mybucket already exists.
 
  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
 
  Your Environment Information -----------------------------
     OS:                 darwin
     Node Version:       4.3.2
     Serverless Version: 1.2.1

Is it something I am missing?

@pmuens

This comment has been minimized.

Member

pmuens commented Nov 25, 2016

Thanks for that @jgardezi 👍
I just read through the thread and noticed that it's currently not possible through CloudFormation 😞 .

Maybe the plugin author / plugin users can chime in here and help?!

@jgardezi

This comment has been minimized.

jgardezi commented Nov 25, 2016

No worries @pmuens

I have already opened the issue matt-filion/serverless-external-s3-event#2 on the plugin project page.

@pmuens

This comment has been minimized.

Member

pmuens commented Nov 25, 2016

Awesome! Thanks for doing that @jgardezi 👍

This makes it easier for others to resolve this issue if they find it here!

@jthomerson

This comment has been minimized.

Contributor

jthomerson commented Dec 14, 2016

@flomotlik I feel like we need this support, but for a different reason than most of the other commenters and original requestor. They were coming at it mostly from the angle of "I don't want to / can't / whatever / use CloudFormation." I'm not coming at it from that angle.

For me, I want to be able to have one microservice (serverless service) that owns a bucket and performs operation X on the bucket, and another microservice (a seperate serverless service) that performs operation Y on the bucket. So the bucket is created in one service - but events on that bucket are subscribed to by multiple services.

It's a similar concept to why I needed to create the https://github.com/silvermine/serverless-plugin-external-sns-events/ plugin - in Serverless if you subscribe to an SNS event, Serverless tries to create the SNS topic, but that's not always desired.

I think in general Serverless will need to allow for subscribing to events of "existing" infrastructure - whether that's (very unfortunately) manually created like some of the other commenters, or whether that's cross-stack subscriptions.

Speaking of cross-stack subscriptions, maybe support could at least be added for cross-stack resource event subscriptions using Fn::ImportValue (as described here)?

@kennu

This comment has been minimized.

Contributor

kennu commented Dec 14, 2016

We are also moving more to the direction of multiple CloudFront stacks, so that everything belongs to a stack but not all stacks are managed by Serverless (we don't want to delete data stacks when upgrading a service to 1.0 or 2.0). So I think that for new services, cross-stack event subscriptions are the thing.

In addition to that, there are of course many cases where buckets already exists outside CloudFormation (e.g. existing Kinesis Firehose or Elastic Transcoder setups) and it will be necessary to somehow subscribe to those events.

@ac360

This comment has been minimized.

Member

ac360 commented Dec 14, 2016

@kennu Thx for the feedback. FYI - working on cross-service referencing is a big priority of ours right now. If you have some thoughts on how you would like this work, please share them in our issues here or in Slack.

@XBeg9

This comment has been minimized.

XBeg9 commented Jan 28, 2017

I got same issue here and found this article: https://aws.amazon.com/blogs/compute/fanout-s3-event-notifications-to-multiple-endpoints/
What AWS suggest us is to use single SNS notification when objectCreated and do subscribers (our lambdas). Is there a way to create S3 Resource that automatically will send message to SNS topic? Can't find any info about that.

@velulev

This comment has been minimized.

velulev commented Jan 29, 2017

For others who are visiting this link, the plug-in seems to have been patched in the last few days, and I can confirm that it works for me, as of 29-Jan-2017

https://github.com/matt-filion/serverless-external-s3-event

Follow the steps as mentioned in the github page, but you can only do step 2 after deployment of the function (unless your function already exists),

so the steps are,

  1. sls deploy

  2. aws lambda add-permission --function-name FUNCTION_NAME --region us-west-2 --statement-id ANY_ID --action "lambda:InvokeFunction" --principal s3.amazonaws.com --source-arn arn:aws:s3:::BUCKET_NAME --source-account YOUR_AWS_ACCOUNT_NUM

  3. sls s3deploy

@XBeg9

This comment has been minimized.

XBeg9 commented Jan 31, 2017

@velulev why not use SNS?

@velulev

This comment has been minimized.

velulev commented Feb 2, 2017

Hi @XBeg9, SNS is a better option if the project uses that service too, but that may not be the case for everyone, which is the case for me too. With those limitations, I have shared my inputs, so that it may help someone in a similar situation to me :)

@milancermak

This comment has been minimized.

milancermak commented May 5, 2017

@velulev FWIW, the add-permission call can be replaced by the following configuration in serverless.yml

resources:
  Resources:
    LambdaInvokePermission:
      Type: AWS::Lambda::Permission
      Properties:
        Action: lambda:InvokeFunction
        FunctionName: service-stage-myLambdaFunctionName
        Principal: s3.amazonaws.com
        SourceArn: arn:aws:s3:::my-bucket
@oyeanuj

This comment has been minimized.

oyeanuj commented Jul 5, 2017

Also coming here with a similar usecase as @jthomerson. One service uploads to an existing bucket that I want serverless to be triggered on put events. @jthomerson how did you end up solving for it?

@ac360 @pmuens Is this possible now? If not, what is your recommendation for such a usecase and is this feature on your radar?

Thank you!

@pmuens

This comment has been minimized.

Member

pmuens commented Jul 6, 2017

@ac360 @pmuens Is this possible now? If not, what is your recommendation for such a usecase and is this feature on your radar?

Heu @oyeanuj thanks for your comment 👍!

Just looked into the code and it looks like the s3 event doesn't support arn strings just yet (s3 is the oldest event source).

However we have #3212 in the making where we're planning to add support for arn detection to all event sources. This way you could reference S3 buckets based on arns with the help of e.g. the Serverless Variables system to make it more flexible.

You could export your bucket in the other stack as an Output and use Fn::ImportValue in combination with the solution @milancermak wrote here: #2154 (comment)

That should work as well.

@oyeanuj

This comment has been minimized.

oyeanuj commented Jul 6, 2017

@pmuens Great, looking forward to the feature! It'd be super helpful to have an example to go for this usecase, if possible. Thank you!

@pmuens

This comment has been minimized.

Member

pmuens commented Jul 7, 2017

Great to hear that @oyeanuj 👍

Here's a comment / thread which could be helpful: #3257 (comment).

You could use smth. like this but leave out the S3 bucket resource and only use the S3 bucket permission resource in your resources section.

Let us know if you need anything else!

@nbk11kk

This comment has been minimized.

nbk11kk commented May 24, 2018

what is the "sls s3deploy" equivalent in yarn
I tried running yarn run s3deploy but it throws unknown command err

@benswinburne

This comment has been minimized.

benswinburne commented May 24, 2018

@nbk11kk Yarn runs commands listed in the scripts section of your package.json. You can add Serverless commands to your scripts. For example you may have

"scripts": {
  "s3deploy": "sls s3deploy"
}

Which you could then run with yarn s3deploy.

If you haven't installed Serverless globally and wish to call the command using npms bin directory

node node_modules/.bin/serverless s3deploy
@nbk11kk

This comment has been minimized.

nbk11kk commented May 24, 2018

My scripts file below

"scripts": {
"deploy": "serverless deploy --stage dev --verbose",
"predeploy": "yarn run build",
"build": "cd Build_Working && build.bat ETL",
"serverless": "serverless",
"invoke-etl": "serverless invoke -f ETL -d "{}" --log",
"invoke": "yarn run invoke-etl",
"deploy_function": "serverless deploy function --function ETL",
"predeploy_function": "yarn run build",
"s3deploy": "sls s3deploy",
"postdeploy": "yarn run s3deploy",
"remove": "serverless remove --stage dev --verbose"
}

error

$ sls s3deploy
Serverless Error ---------------------------------------
Serverless command "s3deploy" not found. Did you mean "deploy"?

@benswinburne :Am I doing anything wrong ??

@benswinburne

This comment has been minimized.

benswinburne commented May 24, 2018

You've installed this plugin with yarn and added serverless-external-s3-event to the plugins array in your serverless.yml file?

@nbk11kk

This comment has been minimized.

nbk11kk commented May 24, 2018

Yes I have installed and added to my .yml file

plugins:
  serverless-plugin-existing-s3
functions:
  ETL:
    handler: TP_S3_to_Redshift.lambda_handler
    package:
      artifact: ETL/artifacts/ETL.zip
    events:
      - existings3: 
          bucket: xyz
          event: s3:ObjectCreated:*
          rules:
            - prefix: logs/LAMBDA_INVOKE_TEST/
@benswinburne

This comment has been minimized.

benswinburne commented May 24, 2018

Not sure that syntax for plugins is correct. Try this

plugins:
  - serverless-plugin-existing-s3
@nbk11kk

This comment has been minimized.

nbk11kk commented May 24, 2018

@benswinburne : Thanks for the tip ..now it ran without any issues.But the trigger didn't get registered on AWS console when I open the lambda function.Should I set it up manually again??

@jordanboston

This comment has been minimized.

jordanboston commented Oct 15, 2018

Just to add to the above, I'm using serverless-plugin-existing-s3 on the current 2.0 branch and it is working great so far.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment