Series of different errors preventing deployment

[murtyjones]

This is a Bug Report

Description

For bug reports:

• What went wrong?

My stack that has been deploying normally for months has started to produce a number of bizarre errors. Some of these have open issues (see links below) but as I'm experiencing all of them and they may be related, I thought I would aggregate the issues.

I'm at the point where I need to delete my stack every time I want to deploy.

• What did you expect should have happened?

I expected my stack to successfully deploy as it has for several months.

• What was the config you used?

I'm deploying to AWS Lambdas using the following framework/plugins:

"serverless": "1.27.3",
"serverless-domain-manager": "^2.4.0",
"serverless-plugin-tracing": "^2.0.0",
"serverless-plugin-warmup": "^3.4.0-rc.1",
"serverless-webpack": "^5.1.5",

• What stacktrace or error message from your provider did you see?

I'm getting one of these three errors every time I deploy:

[2]   Serverless Error ---------------------------------------
[2]  
[2]   An error occurred: ApiLambdaFunction - The function could not be updated due to a concurrent update operation. (Service: AWSLambda; Status Code: 409; Error Code: ResourceConflictException; Request ID: 0c0d332f-65a9-11e8-88b2-ff34b22fa099).
[2]  
[2]   Get Support --------------------------------------------
[2]      Docs:          docs.serverless.com
[2]      Bugs:          github.com/serverless/serverless/issues
[2]      Issues:        forum.serverless.com
[2]  
[2]   Your Environment Information -----------------------------
[2]      OS:                     linux
[2]      Node Version:           8.9.4
[2]      Serverless Version:     1.27.3

[0]   Serverless Error ---------------------------------------
[0]  
[0]   An error occurred: ApiLambdaVersionzxSrUmhVDohdGOvyVjSxLmXvBIyXbQc8Kloi7ttpk4 - A version for this Lambda function exists ( 10 ). Modify the function to create a new version..
[0]  
[0]   Get Support --------------------------------------------
[0]      Docs:          docs.serverless.com
[0]      Bugs:          github.com/serverless/serverless/issues
[0]      Issues:        forum.serverless.com
[0]  
[0]   Your Environment Information -----------------------------
[0]      OS:                     linux
[0]      Node Version:           8.9.4
[0]      Serverless Version:     1.27.3

[0]   Serverless Error ---------------------------------------
[0]  
[0]   Stack:arn:aws:cloudformation:us-east-1:757879768810:stack/auth-service-dev/373520a0-6475-11e8-9cba-50d5cd16c68e is in UPDATE_ROLLBACK_FAILED state and can not be updated.
[0]  
[0]   Get Support --------------------------------------------
[0]      Docs:          docs.serverless.com
[0]      Bugs:          github.com/serverless/serverless/issues
[0]      Issues:        forum.serverless.com
[0]  
[0]   Your Environment Information -----------------------------
[0]      OS:                     linux
[0]      Node Version:           8.9.4
[0]      Serverless Version:     1.27.3

Similar or dependent issues:

• Serverless deploy fails after update from 1.24.1 to 1.25.0 #4631
• 'Concurrent update operation' error for multi-function service results in both deployment and rollback failure.  #4964

Additional Data

• Serverless Framework Version you're using:
  1.27.3
• Operating System:
  Linux

[whitechdevs]

Experiencing the same issue. We have one api and 4 lambda functions as part of the stack.

[murtyjones]

FYI: Deleting my stack in CloudFormation and the serverless buckets in S3 and then deploying as normal ceased my errors. Probably not a permanent fix but enabled me to deploy again. It will obviously cause downtime.

[Centine]

I've had the same issue with "A version [...] already exists".

It didnt replicate on the next try though (where I had enabled debug output)

CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Version - UploadReact[...] 
CloudFormation - CREATE_FAILED - AWS::Lambda::Version - UploadReact[...] 
CloudFormation - UPDATE_ROLLBACK_IN_PROGRESS - AWS::CloudFormation::Stack - [...]-dev
[snipped]
Serverless: Operation failed!

  Serverless Error ---------------------------------------

  An error occurred: UploadReact[...] - A version for this Lambda function exists ( 100 ). Modify the function to create a new version..

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com

  Your Environment Information -----------------------------
     OS:                     darwin
     Node Version:           10.9.0
     Serverless Version:     1.30.1

[StoyanIvanovI]

We had a similar failure to #5285 that has to do with changing reservedConcurrency: 1. We had that particular lambda running on a schedule every minute and deployment with the change above resulted in:

Serverless: Updating Stack...
Serverless: Checking Stack update progress...
................
Serverless: Operation failed!

Serverless Error ---------------------------------------

An error occurred: PollLambdaFunction - The function could not be updated due to a concurrent update operation. (Service: AWSLambdaInternal; Status Code: 409; Error Code: ResourceConflictException; Request ID: [REDACTED]).
 
Get Support --------------------------------------------
   Docs:          docs.serverless.com
   Bugs:          github.com/serverless/serverless/issues
   Issues:        forum.serverless.com

Your Environment Information -----------------------------
   OS:                     linux
    Node Version:           9.11.2
    Serverless Version:     1.33.2

The workaround we managed to do is:

1. Manually continue the Rollback for the CloudFormation stack.
2. Disable the Cloudwatch Event Rule for the Lambda.
3. Deploy via serverless
4. Enable the Cloudwatch Event Rule for the Lambda.

Consecutive deployments with the same configuration complete successfully. I cannot guarantee that there was a Lambda actually executing during the failed deployment but the fact that there were no Lambdas executing during the successful deployment leads me to the conclusion that it has something to do with restricting the reservedConcurrency while Lambdas are being executed.

[pmuens]

Thanks for opening @murtyjones 👍

Seems like this issue has been resolved in the meantime. We've also added some fixes and published some new versions. Feel free to re-open if this issue is still around.