[ECS] [CodeDeploy] [CloudFormation]: CloudFormation support for BLUE/GREEN deployments on ECS

[ilyasotkov]

The feature was announced in November: https://www.youtube.com/watch?v=01ewawuL-IY

For blue/green deployments, AWS CloudFormation supports deployments on AWS Lambda compute platforms only.

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-codedeploy-deploymentgroup.html

Not exactly sure whether AWS::ECS::Service.DeploymentConfiguration would be affected as well.

[ghost]

would be good to get more info on this and what are the plans.

[ngamradt-turner]

We would like to start experimenting with this feature, but without CloudFormation support, it is not practical for us to use.

[ghost]

We would like to start experimenting with this feature, but without CloudFormation support, it is not practical for us to use.

we have been promised that CFN will be getting 1st class treatment going forward ... we shall see :)

[deleugpn]

@steven-cuthill-otm who promised that?

[ghost]

@steven-cuthill-otm who promised that?
jeff barr, have a read ...
https://www.reddit.com/r/aws/comments/avx449/a_quick_cloudformation_update/

[deleugpn]

I read that thread and didn't see any moment the promise to get CloudFormation support for resources as 1st class treatment. The wording says they'll open their roadmap to allow us to tell what we would like to prioritize, which for me does not translate as getting 1st class treatment.

[sarojgharat]

Any update on this ?

[mmoulton]

I'm also very interested in this capability. Specifically I'd like to use it with CDK (aws/aws-cdk#2056)

[angusfz]

Waiting for CFN support

[nitzanyemal123]

+1 For CF support

[mrbueno92]

+1 For CF support

[sarojgharat]

Any update on this?

…

On Wed, May 8, 2019, 15:07 Mauricio Bueno ***@***.***> wrote: +1 For CF support — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#130 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AG7GOJFC4I55K4LUQYWREI3PULF7HANCNFSM4GSAGLIQ> .

[andrewhiles]

Do we know if there's been any movement with this?

[JoeAlamo]

This would be a killer feature to have.

[paddie]

Yep, blocked by this as well. I think that this article summarises pretty clearly why this is not a trivial thing for CF to handle:

It turns out that there’s actually a very good reason why this is not a supported configuration. CodeDeploy is going to be referring to an Auto Scaling Group, and it’s likely that you have defined that ASG in your CloudFormation template, and used !Ref to link the deployment group to it.

The problem comes when you do your first blue-green deployment. CodeDeploy does a blue-green deployment by cloning the ASG to make a new one with the same parameters, but whose instances run the newer version of your application. Once the new version is deployed and healthy, it winds down the old ASG. Then, it updates the deployment group configuration to point to the new ASG, and deletes the old ASG.

This is fine until you go back to your CloudFormation template and need to make a change that touches the ASG. CloudFormation will try to update the ASG that it created, only to find it doesn’t exist any more. Boom, template update failure and rollback.

[ScOut3R]

In regards to ECS blue/green deployments CodeDeploy uses two already existing Target Groups to switch traffic back and forth. While the referenced article holds correct information it does not relate to ECS blue/green deployments but EC2 based ones.
I have a service where I deployed almost everything via CloudFormation, including the two Target Groups used for blue/green deployments. I provisioned the CodeDeploy side with Terraform and everything works quite neatly. Since CodeDeploy does not destroy/clone/recreate the Target Groups both parties are happy.
Apart from capacity concerns as a consumer I don't see any blockers for ECS support.

[okram999]

@ScOut3R - how do the service know that the deployment type is bluegreen, if you configure them using cloudformation. The option to specify deployment type as bluegreen is what where the challenge starts. no?

[ScOut3R]

@okram999 The blue/green CodeDeploy configuration is deployed by Terraform. I pass in as input variables the ECS Service and the two Target Groups (which are built by CloudForamtion) to Terraform which builds the CodeDeploy and CodePipeline configuration.

[noahxp]

+1 For CF support

[pferretti]

+1 For CF support

[ngamradt-turner]

Is there an update on this feature?

[mwarkentin]

Probably helpful to add your votes and follow on the issues on the cloudformation roadmap:

• AWS::CodeDeploy::DeploymentGroup-EcsServices aws-cloudformation/cloudformation-coverage-roadmap#56
• AWS::CodeDeploy::DeploymentGroup-DeploymentType Support BLUE_GREEN also for ECS/Fargate aws-cloudformation/cloudformation-coverage-roadmap#37

[dslove]

I need this too.

[ramanasatyavolu]

Is there a feature provided in CloudFormation to launch blue green deployment through ECS

[bubeamos]

Hi @coultn I see that there a property value CODE_DEPLOY in the "DeploymentController:" property docs here does this mean that this is now available ? And this issue need to be updated ?

[mwarkentin]

Also hoping that this will work with the new CodeDeploy canary deployment strategies!

#229

[richardmaltais]

@paddie For that specific matter, using Blue-Green with CodeDeploy on EC2 instances, we used a custom setup to keep the original ASG, then CodeDeploy-generated ASGs are switched over as usual, drained when needed and the updated LaunchConfiguration is also "copied" over through a Lambda, so that CloudFormation doesn't freak out when comes the time to update it. However, it comes with a few limitations:

• not possible to change the capacity of the ASG (the CodeDeploy-generated one)
• we have to have set the capacity to 0 when applying changes to the LaunchConfiguration and the ASG. Then, the Lambdas do the rest

However, we didn't try to experiment with resource import for the CodeDeploy generated ASG.

[tuttlem]

+1 For CF support

[KiamarzFallahi]

We are excited to announce you can now use AWS CloudFormation to perform Amazon ECS blue/green and canary deployments through AWS CodeDeploy. Blue/green deployments are a safe deployment strategy provided by AWS CodeDeploy for minimizing interruptions caused by changing application versions.

To learn more visit our announcement and the user guide.

[sethstone]

I've been tracking this ticket for a few months and I was excited to see it close, however, the solution provided did not solve the problem I expected. It seems that the new capability allows one to use Cloudformation as the deployment controller - in particular you would be monitoring the deployment in Cloudformation rather than the CodeDeploy console and release changes through stack updates. This may be exactly what others were looking for, but I was expecting was to have the ability to create a deployment group (with CF) using the ECS BlueGreen deployment type and be able to manage the deployment process (once created by CF) in the CodeDeploy console.

Here's an example that I hoped would work after this change

 DeploymentGroup:
    Type: AWS::CodeDeploy::DeploymentGroup
    Properties: 
      ApplicationName: !Ref CodeDeployApplication
      AutoRollbackConfiguration: 
        Enabled: true
        Events:
          - DEPLOYMENT_FAILURE
      BlueGreenDeploymentConfiguration: 
        DeploymentReadyOption: 
          ActionOnTimeout: CONTINUE_DEPLOYMENT
          WaitTimeInMinutes: 0
        TerminateBlueInstancesOnDeploymentSuccess:
          Action: TERMINATE
          TerminationWaitTimeInMinutes: 5
      DeploymentConfigName: CodeDeployDefault.ECSAllAtOnce
      DeploymentGroupName: !Ref CodeDeployDeploymentGroupName
      DeploymentStyle: 
        DeploymentOption: WITH_TRAFFIC_CONTROL
        DeploymentType: BLUE_GREEN
      LoadBalancerInfo: 
        TargetGroupPairInfoList: 
          TargetGroups:
            - Name: !Sub ${ProjectPrefix}-tg1
            - Name: !Sub ${ProjectPrefix}-tg2
          ProdTrafficRoute: 
            ListenerArns: !Ref ALBProductionListener
      ServiceRoleArn: !GetAtt CodeDeployServiceRole.Arn
      ECSServices:
        - ServiceName: !Sub ${ProjectPrefix}-service
        - ClusterName: !Sub ${ProjectPrefix}-ecs-fargate-cluster

Based on the Youtube video I thought this is what we were trying to accomplish because it shows several Deploy stages with ECS Blue/Green Deploy (not a Cloudformation deploy).

Could someone let me know if this is an issue that is being worked on on the roadmap and/or am I doing something wrong to expect this to work? (I based this CF template on a working CLI command that I have)

Thanks!

PS: The error I get is "Encountered unsupported property BlueGreenDeploymentConfiguration"

[KiamarzFallahi]

@sethstone You are correct. This feature does not support creating and configuring a CodeDeploy deployment group to naively perform blue/green deployments for ECS. That is a coverage gap being tracked: aws-cloudformation/cloudformation-coverage-roadmap#37

[clareliguori]

That issue was also closed:
aws-cloudformation/cloudformation-coverage-roadmap#37 (comment)

@sethstone A new issue was created on the CloudFormation roadmap here requesting coverage in AWS::CodeDeploy::DeploymentGroup for ECS blue-green, it would be great for you to comment there with your example template snippet. Thanks!
aws-cloudformation/cloudformation-coverage-roadmap#483

[mwarkentin]

There are a couple of limitations which mean that this feature won't work with our ECS platform (convox):

• Declaring output values or importing values from other stacks is not currently supported for templates defining blue/green ECS deployments.
• You cannot use the AWS::CodeDeploy::BlueGreen hook in a template that includes nested stack resources.
• You cannot use the AWS::CodeDeploy::BlueGreen hook in a nested stack.

Are there plans / work in progress to handle these use cases? Or for more complex use cases like this would it be meant to be handled by the CodeDeploy DeploymentGroup support described above?

[Lasim]

Thanks for the analysis @mwarkentin.

it's meant to me that there is no sense yet to use this feature if these functions are not implemented.

[vinay-nadig-0042]

I see that the issue has been closed since the Cloudformation support has been announced. But, as @mwarketin pointed it out, the limitations are quite crippling. Is there a separate issue where these limitations and their fixes are tracked?

[zzenonn]

I see that the issue has been closed since the Cloudformation support has been announced. But, as @mwarketin pointed it out, the limitations are quite crippling. Is there a separate issue where these limitations and their fixes are tracked?

I opened this issue on the CloudFormation roadmap

aws-cloudformation/cloudformation-coverage-roadmap#483

[simi-obs]

June 24th 2022.

Does anybody know how to control blue/green ECS deployment using Cloudformation(CF) without all the stupid limitations that the hook imposes (no outputs, no dynamic references, no other updates ...)?

I mean at this point I am OK with implementing my own custom resource or even custom resource type. I am willing to bend and punish CF in ugly ways, the only thing I want is a way to perform ECS blue/green deployment as a part of CF deployment (update) process without all the stupid limits.

Edit: I was thinking about implementing custom Cloudformation resource type. This resource would be in control of performing the deployment (lets call it "control resource"). It would use aws code deploy sdk to control the deployment. Do you think it would be possible, or do you see any obstacles?

What I am worried about is erroneous cases when for example the stack update is interrupted due to something and the "control resource" is in progress. I am not sure how does Cloudformation behave in such case (will the "control resource" be notified that the stack update was interrupted?). And there are more cases like that. I guess this implementation would require deep dive into Cloudformation's internal functioning, which might be pretty exhausting. And it still might end up being dead-end.

[teekennedy]

@simi-obs I can assure you that using a custom resource as a workaround for this is not a dead end. I implemented one myself (my very first lambda actually) back in 2019 when I started watching this issue and it's been used thousands of times over in production with a high success rate.

I unfortunately can't post or give you the code without express permission from my employer, but I'll try to give you the rundown of the overall process and the gotchas / lessons learned along the way.

In 2019, it was not possible to create a CloudFormation custom resource that was responsible for just the deployment itself (the "control resource" as you call it), because some the configuration required to create a CodeDeploy enabled ECS service was not allowed by the existing AWS::ECS::Service resource type. I ended up replacing the ECS service resource itself with my custom resource lambda. In addition to managing the ECS service, it managed the CodeDeploy application, and deployment group resources. Whether that was required or just made my life easier I can't recall specifically anymore.

I suggest you approach writing your lambda on an event-by-event basis:

• The create and delete events are likely no-ops unless you end up managing other resources.
• The update event should:
  • Use some kind of locking or retry mechanism to avoid creating a new deployment while one is already in progress. I ended up enforcing that all deployments in the CI/CD pipeline go through CloudFormation to try to prevent this situation, and then wrote a simple polling loop to wait for any in-progress deployments to finish before initiating a new one as a fallback measure. There are better ways, but this solution has yet to cause issues so it is still in use.
  • Collect the parameters for the deployment from the properties of the resource, including at least partially generating the app spec for the deployment.
  • Create the deployment, then wait for it to enter the Failed or Succeeded status.
  • Return the corresponding error or success signal to CloudFormation.

The caveats to watch out for:

• Make sure you send a response to CloudFormation in every scenario. Not sending a response means waiting for CloudFormation to time out against the custom resource, which IIRC defaults to 4 hours. If you end up in this situation, there's nothing you can do but wait, so do everything you can to avoid it. Catch any exceptions and return an error response. Send an error response if the lambda is about to time out. Carefully review the logic used to form and send the response to ensure it always sends something.
• As mentioned above, make sure you handle the use case of overlapping deployments gracefully. The details of this depend on how the deployments fit into the overall release process.
• Deployments can run for much longer than lambdas. There are some CloudFormation custom resource libraries that chain multiple lambda executions together using continuation passing lambda events to work around this. You could also stop the in-progress deployment if the lambda is getting too close to timing out.

Hope this helps!

[simi-obs]

@teekennedy Very nice breakdown, thank you.
Some of the caveats you mentioned have solutions nowadays:

• I.e. instead of using the Lambda backed custom resource, I will implement Cloudformation custom resource type, which does not have some of the limitations (done that in the past).

Regarding the overlapping deployments:

• We are using isolated environments, which are updated only using Cloudformation. What I intend to do is to make sure(enforce) that if the Cloudformation stack update is not ongoing, NO code deploy deployment (of the stack's services) is ongoing. This is in line with what you have mentioned: I will make sure I always wait for the code deploy deployment to either Fail or Succeed.
  If I can enforce this, I should be able to enforce no overlapping deployments (since by default there cannot be overlapping Cloudformation stack updates).

Also I believe I will still have to handle the ECS:Service as a part of the custom resource. Otherwise, I might run into the problem with Cloudformation trying to replace the service (which would probably end up in error) once the TaskDefinition changes.

Overall, thank you for all the tips, will let the community here know how I ended up.