Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Serverless deploy fails after update from 1.24.1 to 1.25.0 #4631

Closed
ihorfito opened this issue Jan 5, 2018 · 28 comments

Comments

Projects
None yet
@ihorfito
Copy link

commented Jan 5, 2018

This is a Bug Report

Description

Serverless application works on 1.24.1 version, after updating serverless to 1.25.0, "serverless deploy" fails with error. After changing serverless version back to 1.24.1 it works again.
For bug reports:

  • What went wrong?
    Deploy fails after changing serverless version from 1.24.1 to 1.25.0

  • What did you expect should have happened?
    Deploy should run without errors after serverless upadte

  • What stacktrace or error message from your provider did you see?

Additional Data

  • Serverless Framework Version you're using: 1.25.0
    Serverless Error ---------------------------------------
    An error occurred: MyLambdaVersionl7sH2kLfxNHEEk4pJj557EN0svGZitSVdsF8sksyz1I - A version for this Lambda function exists ( 53 ). Modify the function to create a new version..

  • Stack Trace --------------------------------------------
    ServerlessError: An error occurred: CollectLambdaVersionl7sH2kLfxNHEEk4pJj557EN0svGZitSVdsF8sksyz1I - A version for this Lambda function exists ( 53 ). Modify the function to create a new version.. at provider.request.then (/usr/local/lib/node_modules/serverless/lib/plugins/aws/lib/monitorStack.js:112:33) From previous event: at AwsDeploy.monitorStack (/usr/local/lib/node_modules/serverless/lib/plugins/aws/lib/monitorStack.js:26:12) at provider.request.then (/usr/local/lib/node_modules/serverless/lib/plugins/aws/lib/updateStack.js:84:30) From previous event: at AwsDeploy.update (/usr/local/lib/node_modules/serverless/lib/plugins/aws/lib/updateStack.js:84:8) From previous event: at AwsDeploy.BbPromise.bind.then (/usr/local/lib/node_modules/serverless/lib/plugins/aws/lib/updateStack.js:101:12) From previous event: at AwsDeploy.updateStack (/usr/local/lib/node_modules/serverless/lib/plugins/aws/lib/updateStack.js:95:8) From previous event: at AwsDeploy.BbPromise.bind.then (/usr/local/lib/node_modules/serverless/lib/plugins/aws/deploy/index.js:135:39) From previous event: at Object.aws:deploy:deploy:updateStack [as hook] (/usr/local/lib/node_modules/serverless/lib/plugins/aws/deploy/index.js:131:10) at BbPromise.reduce (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:368:55) From previous event: at PluginManager.invoke (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:368:22) at PluginManager.spawn (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:386:17) at AwsDeploy.BbPromise.bind.then (/usr/local/lib/node_modules/serverless/lib/plugins/aws/deploy/index.js:101:48) From previous event: at Object.deploy:deploy [as hook] (/usr/local/lib/node_modules/serverless/lib/plugins/aws/deploy/index.js:97:10) at BbPromise.reduce (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:368:55) From previous event: at PluginManager.invoke (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:368:22) at PluginManager.run (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:399:17) at variables.populateService.then (/usr/local/lib/node_modules/serverless/lib/Serverless.js:102:33) at runCallback (timers.js:637:20) at tryOnImmediate (timers.js:610:5) at processImmediate [as _immediateCallback] (timers.js:582:5) From previous event: at Serverless.run (/usr/local/lib/node_modules/serverless/lib/Serverless.js:89:74) at serverless.init.then (/usr/local/lib/node_modules/serverless/bin/serverless:42:50)

@HyperBrain

This comment has been minimized.

Copy link
Member

commented Jan 5, 2018

This is a duplicate of #4599 .

There seems to be one very rare condition where this happens once, but works afterwards. Can you try the 2 steps I mentioned here: #4599 (comment) ?

However, to catch the root reason, do you have a detailed CloudFormation update log available? Especially the order of the resource modifications would be helpful.

@ihorfito

This comment has been minimized.

Copy link
Author

commented Jan 5, 2018

@HyperBrain Yes, I have access to CloudFormation update log,
Some functions are updated correctly, and some fails when I run "serverless deploy"(v1.25.0).
I am using "serverless-webpack": "4.2.0" and after source changes deploy is successful.

@HyperBrain

This comment has been minimized.

Copy link
Member

commented Jan 5, 2018

Good. Can you check if, for the failing functions in the CF logs, the new version resources are created before the old ones are removed or if there is another obvious CF update order problem? My guess is, that in some cases when updating from 1.24.1 to 1.25.0 the implicit order is wrong because Serverless misses to add a vital dependency to its resources (which would be the root case - hidden until 1.25.0, because the version resource logical ids were wrong in case, only the configuration changed).

@ihorfito

This comment has been minimized.

Copy link
Author

commented Jan 5, 2018

@HyperBrain
here is CloudFormation Log
https://gist.github.com/ihorfito/fcf1ceeecaff820012f9eea148d8f06d

  1. CloudFormation - UPDATE_IN_PROGRESS - AWS::Lambda::Function
  2. CloudFormation - UPDATE_COMPLETE - AWS::Lambda::Function
  3. CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Version
  4. CloudFormation - CREATE_FAILED (or some logs has CREATE_COMPLETE) - AWS::Lambda::Version
@daniel-cottone

This comment has been minimized.

Copy link
Contributor

commented Jan 10, 2018

Running into this problem as well; reverting to 1.24.1 fixes.
Also using serverless-webpack and serverless-aws-alias.

@horike37 horike37 added the bug label Jan 10, 2018

@horike37 horike37 added this to the 1.26 milestone Jan 10, 2018

@HyperBrain

This comment has been minimized.

Copy link
Member

commented Jan 11, 2018

I think I found the problem with the not automatically deleted old versions if a new one is deployed.
The reason is, that the version resources are set to RETAIN, i.e. if a function configuration is changed and a new version is created but no new function code, the replaced one does not get deleted.
As there is only one version for the same code allowed, this errors in that case.
We should set the version's DeletionPolicy to DELETE and it should work. However, side-effects should be evaluated first.
@daniel-cottone Your problem could be related to the alias plugin, as I'm not sure if the current version handles this case correctly. I'll have to check and update the plugin accordingly.

@l-hendriks

This comment has been minimized.

Copy link

commented Jan 23, 2018

+1 We also encounter this issue, but we don't use the alias plugin (but do use serverless-webpack)

@horike37 horike37 modified the milestones: 1.26, 1.27 Jan 29, 2018

@ozbillwang

This comment has been minimized.

Copy link

commented Apr 8, 2018

@HyperBrain

Today I hit this issue as well. Do you talk about the code in this line:

https://github.com/serverless/serverless/blame/2a3d57e3907ecce537bcf8d1b73074ed6a38a1b4/lib/plugins/aws/package/compile/functions/index.js#L424

  cfLambdaVersionTemplate() {
    return {
      Type: 'AWS::Lambda::Version',
      // Retain old versions even though they will not be in future
      // CloudFormation stacks. On stack delete, these will be removed when
      // their associated function is removed.
      DeletionPolicy: 'Retain',
      Properties: {
        FunctionName: 'FunctionName',
        CodeSha256: 'CodeSha256',
      },
    };
  }

Seems this bug has been moved to 1.27, then how to fix it temporarily?

I am not sure what I should do for below steps:

(1) change a function's sources => deploy
(2) change a function's sources and config => deploy
(3) change only a function's config (e.g. memory) => deploy

Can you provide detail steps?

Updates:

Finally I understand what to do. The sources means source codes, environment setting or any thing to make the lambda function to active save button.

@tgfischer

This comment has been minimized.

Copy link

commented May 16, 2018

I ran into this issue today. I am using Serverless ^1.25.0, and serverless-webpack ^5.1.0

@Enase

This comment has been minimized.

Copy link

commented May 16, 2018

@tgfischer do you use serverless-aws-alias? The last release fixes the trouble

@tgfischer

This comment has been minimized.

Copy link

commented May 16, 2018

@Enase Nope. I updated to the latest versions of serverless and serverless-webpack, and I'm not getting the error so far. I'll comment if it happens again

@murtyjones

This comment has been minimized.

Copy link
Contributor

commented May 31, 2018

I'm getting this error with serverless 1.27.3 (sls-webpack 5.1.5).

[0]   Serverless Error ---------------------------------------
[0]  
[0]   An error occurred: ApiLambdaVersionzxSrUmhVDohdGOvyVjSxLmXvBIyXbQc8Kloi7ttpk4 - A version for this Lambda function exists ( 10 ). Modify the function to create a new version..
[0]  
[0]   Get Support --------------------------------------------
[0]      Docs:          docs.serverless.com
[0]      Bugs:          github.com/serverless/serverless/issues
[0]      Issues:        forum.serverless.com
[0]  
[0]   Your Environment Information -----------------------------
[0]      OS:                     linux
[0]      Node Version:           8.9.4
[0]      Serverless Version:     1.27.3
@l-hendriks

This comment has been minimized.

Copy link

commented May 31, 2018

I've learned that you get this issue when you deploy the script using different roles. So if you sls deploy with role A and then sls deploy with role B, and the function code is not changed, you will get the error. If you change the function code or use the role of the previous deployment it works.

@murtyjones

This comment has been minimized.

Copy link
Contributor

commented May 31, 2018

@l-hendriks Thanks for the input. I'm not all that familiar with roles in this context. In your case, did a role change occur outside the scope of your serverless.yml? I haven't modified anything in my config relating to roles so I'm unsure how I ended up in that situation, assuming I have the same error source.

@l-hendriks

This comment has been minimized.

Copy link

commented Jun 2, 2018

In our case it was deploying from multiple sources. For example once on my pc and then in the CI pipeline. But also when a colleague deploys and then I want to deploy as well (each of us have different keys).

@murtyjones

This comment has been minimized.

Copy link
Contributor

commented Jun 3, 2018

That's helpful, thanks.

For anyone else struggling with this, deleting the CloudFormation templates (IE deleting the entire stack) and all S3 deployment buckets got rid of the error for now. Not a great solution for production environments though.

@l-hendriks

This comment has been minimized.

Copy link

commented Jun 3, 2018

That does work, but as you said in a production environment this is not really feasible. Another workaround is to just modify the code a bit. We learned to just ignore the error for our CI pipeline setup. It means that it doesn't deploy when there are no code changes, but as there are no code changes it doesn't have to deploy. When there are code changes, the error is gone.

@murtyjones

This comment has been minimized.

Copy link
Contributor

commented Jun 3, 2018

In my case that techniqu yielded different errors each time I deployed. I've opened up #5016 to illustrate the issues, but for now nuking my stack was the only way to get unstuck.

@jthomerson

This comment has been minimized.

Copy link
Contributor

commented Sep 13, 2018

I was encountering the same error when using my serverless-plugin-cloudfront-lambda-edge plugin. I found the cause and documented it at silvermine/serverless-plugin-cloudfront-lambda-edge#21 and will fix it soon.

That's not the exact cause of the issue encountered in this issue, but perhaps the cause is similar and reading what I wrote on that issue might help someone find the issue here.

@pmuens

This comment has been minimized.

Copy link
Member

commented Feb 8, 2019

Closing since this is a rather old issue. In the meantime we've published many different releases which should address this issue.

Feel free to re-open if this is still a problem.

@pmuens pmuens closed this Feb 8, 2019

@skydroid

This comment has been minimized.

Copy link

commented Mar 11, 2019

Hello! I just encountered this issue with serverless version 1.38.0

I tried sls deploy and sls deploy --force to no avail.

@l-hendriks

This comment has been minimized.

Copy link

commented Mar 12, 2019

You can just change the function code a bit (add a console.log or something) and redeploy. Then when you remove that extra bit and deploy again, it should work as well. I encounter this sometimes when serverless wants to redeploy a lambda function with identical code.

@skydroid

This comment has been minimized.

Copy link

commented Mar 12, 2019

Thanks for your insight. While that is a workaround, I don't think that will work for automated production deployments?

@l-hendriks

This comment has been minimized.

Copy link

commented Mar 12, 2019

Yes it is but it's not really a problem for us anymore. We get this very very seldom, if that happens in the automated deployment, we just add a useless line in production and then remove it again. Not ideal, but it works :p

@skydroid

This comment has been minimized.

Copy link

commented Mar 12, 2019

I put in a useless (commented) line in my lambda function and the issue still persists.

@l-hendriks

This comment has been minimized.

Copy link

commented Mar 12, 2019

try adding a console.log. If you add a commented line and webpack the function, the commented line is stripped out, leaving your code unchanged

@skydroid

This comment has been minimized.

Copy link

commented Mar 12, 2019

Ah! Ok that did. Seemed it kicked it back into a "normal" state. I can deploy at will now.

@l-hendriks

This comment has been minimized.

Copy link

commented Mar 13, 2019

Cool! This always works. Not really a solution, but not sure if this will ever get fixed...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.