Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Skip resources if already exist #3183

Open
marckaraujo opened this Issue Feb 2, 2017 · 59 comments

Comments

Projects
None yet
@marckaraujo
Copy link

marckaraujo commented Feb 2, 2017

This is a (Feature Proposal)

Description

For bug reports:

  • What went wrong?

I had a bug that cloudformation just stuck at UPDATE_ROLLBACK_FAILED.
So I should to delete the stack and deploy serverless again.
But now I got another problem:

Serverless Error ---------------------------------------

    An error occurred while provisioning your stack: AvailableDynamoDbTable
    - Available already exists.
  • What did you expect should have happened?

I think that database is too critical in production level to wont use Retain, In a simple wrong deploy or remove the stack can banish all your tables, wrong deploy can easily rollback but data is really critical.

So I suggest to have something like: serverless deploy --skipResources, so it wlll skip the resources that already exist and cloudformation wont bring that error.

Similar or dependent issues:

Additional Data

  • Serverless Framework Version you're using: 1.6.1
  • Operating System: Mac OS El Capitan
  • Stack Trace:
  • Provider Error messages:

@pmuens pmuens added the feature label Feb 2, 2017

@rowanu

This comment has been minimized.

Copy link
Member

rowanu commented Feb 3, 2017

If the data is too important to delete, you probably shouldn't be managing the Table resource in your service definition - it belongs outside, either in a "resource-only service" (if you want to use sls to manage it), or in a completely different CFN template.

@marckaraujo

This comment has been minimized.

Copy link
Author

marckaraujo commented Feb 3, 2017

As I said, a bug anywhere could happen, as it did for me. So its not a wrong pattern of management. It can and do happen. So this feature will save a lot of headache if this happen in a prod env.

@rowanu

This comment has been minimized.

Copy link
Member

rowanu commented Feb 3, 2017

I don't think this is a bug; You are trying to create a resource with exactly the same name as an existing resource, which is not allowed for DynamoDb Tables (but is allowed for other resources).

Edit: Just thought of another example; For some resources CFN will generate a unique name (by affixing a random string), so how will you know if the resource is to be kept or not? I'm sure there's other edge cases like this that make such a feature complex and error-prone to implement (even though it sounds like a good idea on the surface).

@marckaraujo

This comment has been minimized.

Copy link
Author

marckaraujo commented Feb 3, 2017

This case is not like I am trying to upload a new project with resources already deployed.

Its related to bugs like that: #3146
Even so this AWS Bug persist since 2012.

I dont know if it is applicable to every "resources", atleast for databases I think.

I already have a small project running with sls 0.5.6 and now I decided to create a big one under serverless framework.

Another problem was:

I did a mistake to create a new DynamoDB table with wrong index, so I decided to delete the database and create again. But after that, serverless gives the same error for all others tables, so I needed to delete all my tables and deploy again. (Obs: I use retain in config).

@hermanmedsleuth

This comment has been minimized.

Copy link

hermanmedsleuth commented Feb 8, 2017

I too am having a problem with this.

I have a lambda service which subscribes to a SNS topic created, and written to, by a server resident service. I am attempting to user ServerLess™ to manage this lambda, but I get the following error in deployment:

Serverless: Checking Stack update progress...
................Serverless: Deployment failed!
 
  Serverless Error ---------------------------------------
 
     An error occurred while provisioning your stack: SNSTopicOrderStatusNotifications
     - OrderStatusNotifications already exists.

I understand, from #1842, that ServerLess™ is failing when it attempts to create the topic.

It makes sense to me that the service that writes to the topic should create the topic. Having the subscriber(s) create SNS topics, especially in cases such as mine, where the topic is widely subscribed to, seems sub-optimal. In this case ServerLess™ should skip the creation of the pre-existent resource.

@hermanmedsleuth

This comment has been minimized.

Copy link

hermanmedsleuth commented Feb 8, 2017

Apparently, my situation is already taken care of. I missed this in the docs.

Apologies for the noise.

@pmuens

This comment has been minimized.

Copy link
Member

pmuens commented Feb 8, 2017

@hermanmedsleuth great to see that your issue is resolved 👍

@marckaraujo

This comment has been minimized.

Copy link
Author

marckaraujo commented Feb 13, 2017

@pmuens Another problem related to this:

I created a new DynamoDBTable resource in Serverless.yml and also a stream function to this table.

It gives me an error because stream in this table was not SET, its ok. Since I just created the table.
So, I go to the console, enabled the stream, now tried to deploy serverless again.

And the error becomes again, DynamoDBTable already exists.

I'll keep this post updated with all errors that I found to convince you that it should be implemented. XD

@pmuens

This comment has been minimized.

Copy link
Member

pmuens commented Feb 14, 2017

@marckaraujo thanks for updating! 👍

The stream event has some known bugs which might be related here (see thread here: #2488).

@mariogintili

This comment has been minimized.

Copy link

mariogintili commented Mar 14, 2017

I would be happy for this feature to exists as well

@mariogintili

This comment has been minimized.

Copy link

mariogintili commented Mar 14, 2017

@rowanu

I'm sure there's other edge cases like this that make such a feature complex and error-prone to implement (even though it sounds like a good idea on the surface).

I don't think thats a valid reason not to implement it. Shouldn't the community try to find a collective solution to this?

@HyperBrain

This comment has been minimized.

Copy link
Member

HyperBrain commented Mar 14, 2017

I agree with @rowanu here. If you want to put a further level of safety onto your resources you should put them into a separate resource only CF stack, export the resource names and import them via Fn::ImportValue in your function stack. This also guarantees that the resource stack cannot be deleted as long as it is referenced anywhere.
A CF stack naturally owns its resources and makes sure that everything is created/changed/updated in a transactional way.
BTW: You should not specify a TableName property with DynamoDB tables as this prevents any changes that need Replacement like changing the keys, etc. A better way is to grab the tablename via Ref where you need it and publish it through environment variables to your code.

@hermanmedsleuth

This comment has been minimized.

Copy link

hermanmedsleuth commented Mar 14, 2017

It makes sense, to me, to consider, and handle, trigger resources as exogenous to the CF stack. That a CF stack is expected to manage a resource that incipiates the instantiation of itself, strikes me as poor design choice. A lambda function is not always the entirety of an application. It is (probably most) often a service to a larger application.

@mariogintili

This comment has been minimized.

Copy link

mariogintili commented Mar 14, 2017

@hermanmedsleuth then why is my serverless.yml file an interface to CF and not just a subset of it? It looks like something partially implemented right now

@laardee

This comment has been minimized.

Copy link
Member

laardee commented Mar 14, 2017

@kennu has made a plugin for deploying additional CF stacks with Serverless.

@marckaraujo

This comment has been minimized.

Copy link
Author

marckaraujo commented Mar 15, 2017

@kennu @laardee This plugin is very nice but I dont know if it solves the problem described here. Since if you need to re-deploy a resource like Dynamodb table you will get an error even if it is in another CF stack.

@laardee

This comment has been minimized.

Copy link
Member

laardee commented Mar 15, 2017

@marckaraujo that plugin helps you to manage multiple CF stacks, like @rowanu and @HyperBrain suggested. If your service deployment then fails, the DB and other critical resources won't be affected when your service CF stack needs to be removed.

@nenti

This comment has been minimized.

Copy link

nenti commented May 19, 2017

I am constantly running in this issue aswell. It prevents continuous deployment on my system. I created a stackoverflow discussion on how to handle tables that block updates: http://stackoverflow.com/questions/43771000/how-to-migrate-dynamodb-data-on-major-table-change/43790256#43790256
Maybe that helps you guys to implement a better handling of table deployment in this framework.

@pmuens

This comment has been minimized.

Copy link
Member

pmuens commented May 22, 2017

🤔 I agree with the point that you might want to put this kind of resources into separate stacks and manage them there.

@brianneisler and @eahefnawy what are your thoughts on this?

@hermanmedsleuth

This comment has been minimized.

Copy link

hermanmedsleuth commented May 24, 2017

I'm still not seeing a solution here for defining a pre-existing, possibly shared, resource as a trigger for a lambda, in ServerLess™.

@mariogintili

This comment has been minimized.

Copy link

mariogintili commented May 25, 2017

Given Serverless is a private company Do you guys have any open governance structure for the framework? Like RFCs that will let the Serverless user base tackle discussions like this a bit more formally? cc @pmuens

@pmuens

This comment has been minimized.

Copy link
Member

pmuens commented May 25, 2017

Good question @mariogintili 👍 Thanks for asking!

/cc @brianneisler @ac360 @worldsoup 🔝

@nenti

This comment has been minimized.

Copy link

nenti commented Jun 1, 2017

@pmuens When you agree with putting your databases into another stack. Is that a stack not run by serverless?

From my point of view serverless is a perfect fit for building micro services which by definition have their dedicated resources they require to function and shall not share them with other services. Also by definition running serverless requires not only to manage software but also the required operating systems from code. Thus, managing dynamodb externally tasks like the suggestion to go back to a hosted solution.

@pmuens

This comment has been minimized.

Copy link
Member

pmuens commented Jun 2, 2017

@pmuens When you agree with putting your databases into another stack. Is that a stack not run by serverless?

@nenti Yes, you could do that and then reference this DynamoDB with Cross-Stack-References (via Fn::ImportValue) to tie them together.

Also note that Serverless supports function-free services (#2499) so you could even deploy this DynamoDB only-service via Serverless.

I agree that it's nice to have everything in one CloudFormation stack but you might want to split things up as soon as your service gets bigger and you want more control over the resources.

@hermanmedsleuth

This comment has been minimized.

Copy link

hermanmedsleuth commented Jun 2, 2017

If an event can reference a pre-existing resource, then ServerLess™ should not gork if it cannot create it. However, when the resource is a SNS Topic, that is not the case. This is inconsistent, at best.

@nenti

This comment has been minimized.

Copy link

nenti commented Jun 7, 2017

@pmuens Ok I can put it into another stack. But then when I change my table and redeploy I get same error from serverless. Even if I use a function-free service. Because serverless doesn't handle dynamodb. So If my serverless consists of dynamodb only that doesn't change this fact.

@pmuens

This comment has been minimized.

Copy link
Member

pmuens commented Jun 12, 2017

@pmuens Ok I can put it into another stack. But then when I change my table and redeploy I get same error from serverless. Even if I use a function-free service. Because serverless doesn't handle dynamodb. So If my serverless consists of dynamodb only that doesn't change this fact.

The "best-practice" (if you don't want to put everything in one template and run into the problem this issue describes) would be to deploy the DynamoDB with a separate CloudFormation file and import output values via Fn::ImportValue. That's also something AWS recommends.

This way you'd get highly coupled services, but it's a good thing if you cannot remove the DynamoDB stack when its values are still used in another stack.

@zbuckholz

This comment has been minimized.

Copy link

zbuckholz commented Aug 30, 2017

I'm pretty new to the serverless project, been lurking.
But I use cloudformation almost daily, and was curious if the serverless deploy could have a complimentary argument named 'update'.

Since it's basically just generating a cloudformation template, that template can be applied as an update instead of a deploy.

An update will compute the difference between what is already present and notify you of the changes before taking action.

@pmuens

This comment has been minimized.

Copy link
Member

pmuens commented Aug 30, 2017

@zbuckholz thanks for your comment and welcome abroad 👍

Do you mean CloudFormation change sets? We've spent some time to figure out if we should / could add them, but unfortunately introducing them now would be a breaking change and could therefore only be done in v2.

@jthomerson

This comment has been minimized.

Copy link
Contributor

jthomerson commented Sep 1, 2017

I'm just throwing my hat in the ring on the side of: there's nothing that Serverless needs to implement here. As many have said, you can manage those resources in a separate resource-only stack (I default to this and have tons of stacks that have only S3 buckets / DynamoDB tables / etc). Then your services are made to depend on those by means of imports or just name patterns.

How would Serverless implement a --skip-existing-resources anyway? Serverless is creating a CloudFormation template and letting CloudFormation do the deployment. If Serverless didn't include the resources in the CF template (and they were there in a previous version), CF would remove them.

@dimitrovs

This comment has been minimized.

Copy link

dimitrovs commented Sep 3, 2017

A workaround is to comment-out the resource you know are already in AWS before running deploy. I too, would like to see support for this feature in Serverless. May be it can be done per resource type and Serverless can just look up the resources by their id in AWS to see if they exist before generating the Cloudformation.

@jthomerson @rowanu putting the resources in a separate service may be better design but it adds unnecessary complexity. The simplicity of Serverless over Cloudformation or SAM is a major advantage in my opinion.

@mataide

This comment has been minimized.

Copy link

mataide commented Sep 3, 2017

I also have a problem now that I just hit the cloudformation creation limit, so I cant deploy all my resources at once. Then I need to comment-out the resources already deployed.

But now if I commment-out the resources, my linked dynamodb streams doesnt work anymore.
(If you ask me why?) thats why:

streamRoom:
                  description: "Stream Room"
                  handler: api/stream/room/handler.default
                  memorySize: 256
                  timeout: 36
                  events:
                    - stream:
                        type: dynamodb
                        arn:
                          Fn::GetAtt:
                            - RoomDynamoDbTable
                            - StreamArn
                        batchSize: 1

With this workaround, I dont need to setup streamId, so it works for multiple stages with an easy configuration, but if RoomDynamoDbTable is comment-out. Then it doesnt work. @pmuens

@pmuens

This comment has been minimized.

Copy link
Member

pmuens commented Sep 4, 2017

@dimitrovs @mataide the problem with the "commenting out" strategy is that CloudFormation will see that the resources are not there anymore and hence removes them.

The CloudFormation template always reflects how the actual deployment / state will look like after applying / deploying it. That's why commenting out won't work.

@Saandji

This comment has been minimized.

Copy link

Saandji commented Oct 29, 2017

any progress on this issue?

@marckaraujo

This comment has been minimized.

Copy link
Author

marckaraujo commented Oct 31, 2017

Ok, a lot of people give their opinion but what is the solution for this? Because until now I have any from Serverless, another plugin or workaround.

@OrKoN

This comment has been minimized.

Copy link

OrKoN commented Nov 8, 2017

I think --skip-custom-resources is not a solution because, as said, the CF will delete resources not included in the update. The ultimate solution is to handle database via a separate cloud formation stack. In this case there is still problem that you cannot update the same table with new indexes but this is a known problem and it is not coupled to deployment of lambda functions then.

My only concern right now is that without defining my tables in serverless.yaml I am not able to use the serverless-dynamodb-local plugin. This is what I am looking for right now: running a local dynamodb using a custom CF template.

P.S. I guess I need smth like this https://github.com/steven-bruce-au/dynamodb-local-cloud-formation

@kinmanlam

This comment has been minimized.

Copy link

kinmanlam commented Jan 26, 2018

Hello all,
I am ramping up on Serverless, and have already experienced a need to reference an existing resource as an event trigger to my function.

After thoroughly reviewing this thread, it is apparent that more documentation from Serverless is required to handle everyday scenarios.

For anyone who is just starting with Serverless, the resources section is used to define NEW resources you want to create as part of your services roll out. Only use it to define new resources. If you start using the resources section to reference existing resources you are creating resource versioning issues. This is why the Op suggested -skipResources option would not be a good solution.

Example: You have two serverless.yml files both defining a shared Resource, both with slightly different properties whether on purpose or by accident. Which was is the correct one? Or let say both Resource definitions are the same, but you want to change the properties, now you will have to update the Resource definition in two places. Not a good approach.

The better approach is the have a master or a base serverless.yml define all Resources once, and have a serverless variable or use the Fn:ImportValue in all subsequent serverless.yml to reference the ARN of the existing Resource.

@mafrosis

This comment has been minimized.

Copy link

mafrosis commented Feb 5, 2018

Hello all. Another vote for being able to manage a subset of resources with Serverless.

We have already existing infrastructure managed by terraform - and which is owned by another team. We can't include DynamoDB resources in our serverless.yml at production deploy time, since they already exist. We need these resources defined for offline dev.

@logicminds

This comment has been minimized.

Copy link

logicminds commented Feb 22, 2018

Just another use case. My gitlab CI setup includes a preview build which builds and deploys a serverless stack. Upon branch merge or branch deletion the CI will remove the associated serverless stack. The initial build out works as intended but if additional commits are added to the branch the build pipeline gets triggered and runs sls deploy again which causes the dyamodb table exists failure.

As a workaround I am just removing the table before hand

In this case I don't care about the data but I would have thought the cloud formation would have seen that the dynamodb table existed and moved on.

I suppose I could move just the dynamodb table creation out of serverless into its own CF deploy, but I am lazy ATM.

@ali-himindz

This comment has been minimized.

Copy link

ali-himindz commented Feb 26, 2018

Can this be handled via a plugin? I am thinking to right a plugin that will just remove the resource section from the output? Am I in wrong direction?

@mafrosis

This comment has been minimized.

Copy link

mafrosis commented Feb 26, 2018

@ali-himindz The brute force approach I use removes the entire resource key from serverless.yml at deploy time (before restoring it).

cat serverless.yml | yq -y 'del(.resources)' | sponge serverless.yml
sls deploy
git checkout serverless.yml

(Requires the python yq tool, jq and moreutils).

@berlinguyinca

This comment has been minimized.

Copy link

berlinguyinca commented Apr 26, 2018

that's rather sad, can't we just have a flag, which says 's skip if exists? Shouldn't be too hard...

@jthomerson

This comment has been minimized.

Copy link
Contributor

jthomerson commented Apr 26, 2018

@berlinguyinca what do you want Serverless to do? Serverless is not creating the resources. Serverless simply creates a CloudFormation template and lets CloudFormation create the resources.

@berlinguyinca

This comment has been minimized.

Copy link

berlinguyinca commented Apr 26, 2018

@jthomerson

This comment has been minimized.

Copy link
Contributor

jthomerson commented Apr 26, 2018

@berlinguyinca Sure, Serverless can call any AWS API that it needs to. The point is, though, that this isn't the responsibility of Serverless - Serverless isn't in the business of creating resources. If it added a feature to see if a resource existed, it would need to constantly keep up with the APIs for every service that AWS provides. Serverless' job is to create a CloudFormation template that it sends to CloudFormation so that CloudFormation can do the heavy lifting of integrating with all the AWS services.

I don't understand your usecase where you're defining a resource in Serverless that already exists. What originally created the resource? If something else created it, why is it being defined again in the Serverless template?

@berlinguyinca

This comment has been minimized.

Copy link

berlinguyinca commented Apr 26, 2018

@sernaton

This comment has been minimized.

Copy link

sernaton commented May 25, 2018

Also hitting the problem. Use case: simple service with dynamodb, want to create if not existing, else update or at least ignore, for several environments. We are looking for simplicity - it is a pain to do this by hand, and complicate the deployment with extra steps...

@RogerAlex75

This comment has been minimized.

Copy link

RogerAlex75 commented Jun 17, 2018

Wish I knew what half this shit ment.

@shimont

This comment has been minimized.

Copy link

shimont commented Jul 20, 2018

I am up for "skip if exists"

@qtiki

This comment has been minimized.

Copy link

qtiki commented Jul 24, 2018

I just ran into this issue by accidentally deleting the wrong Serverless app (luckily a dev version from the wrong branch and not the production app). Our DynamoDB tables all have DeletionPolicy: Retain for exactly these situations. However because the table was already there I could not re-deploy the app.

Here's how I worked around the issue:

  1. I deleted the tables manually in AWS with the create backup option selected.
  2. I re-deployed the Serverless app to fix the broken CloudFormation state.
  3. I deleted the tables manually again (no backup this time since they were empty).
  4. I restored all of the backups I made earlier into new tables with the same name as the originals.

This way I was able to retain all the data and got the CloudFormation stack to work properly again. I confirmed this by making a change to the ProvisionedOutput of one of the tables and then deployed, which worked as expected.

I know this is a bit unconventional as you're not supposed to touch CloudFormation controlled resources manually, but I would imagine if this would happen to someone in production environment this workaround might be a real life saver.

And for the feature proposal: I agree with what many have said here that it doesn't make any sense to declare the same resource in multiple Serverless apps. However for my particular case, ie. removing the stack and then later trying to re-deploy it, the feature proposal to fix this for all resource types sounds way too big as the root cause for this is CloudFormation's DeletionPolicy: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-attribute-deletionpolicy.html

I think the only feasible way would be to create custom Serverless plugin for each resource type to migrate the existing data somehow when the stack is re-created (using the backups like I did can be pretty slow as it can take ~4 hours). Something like that could also be used for "branching" apps so you could for example migrate the DynamoDB table data to the new app. And if you start talking about data migration then you might want to think about migrating data when you need to change the table keys or join two tables together etc... (thinking of something like FlyWay but for DynamoDB). And btw by feasible I mean "possible but a huge amount of work!" 😁

@CodySwannGT

This comment has been minimized.

Copy link

CodySwannGT commented Oct 11, 2018

I'll second what @sernaton said. With complimentary services like Amplify, it would be nice to be able to use ServerLess to, say, add a stream to a dynamodb table created using Amplify.

@mataide

This comment has been minimized.

Copy link

mataide commented Nov 23, 2018

@pmuens I think we already many feedbacks to get a conclusion. Anyone could provide a solution?

@brunocascio

This comment has been minimized.

Copy link

brunocascio commented Jan 25, 2019

I am up for "skip if exists"

+1 :(

@tvp-jwheeler

This comment has been minimized.

Copy link

tvp-jwheeler commented Feb 4, 2019

Another option could be to import existing resources.
Allow serverless to hook into existing infrastructure to start working on it.

Terraform has a similar feature:
https://www.terraform.io/docs/import/index.html

@7jpsan

This comment has been minimized.

Copy link

7jpsan commented Mar 20, 2019

Hi guys, just had the same problem and a workaround whilst this is not currently supported, is to use serverless.js (we need yaml-boost to make use of the awesome '<<<' operator which means the object exported can be manipulated).

In AWS we wanted to deploy only one ECR as an additional stack for all environments and now that the parsing happens before, we can manipulate the obj and add.remove keys. Use with caution though.

Full example below:

const path = require('path');
const minimist = require('minimist');
const yaml = require('yaml-boost');

const args = minimist(process.argv.slice(2));
const slsConfig = yaml.load(path.join(__dirname, 'serverless.core.yml'), args);

if(args.stage && args.stage === 'prod'){
  delete slsConfig.custom.additionalStacks["external-ecr"];
}

module.exports = slsConfig;
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.