ECS/Fargate: DesiredCount should be completely optional

[pgarbe]

Tell us about your request
The DesiredCount property should be optional both when creating and updating an ECS service. If it's not defined the default should be the min value of the attached autoscaling (or 0 if there's no autoscaling).
An alternative could be to introduce another property InitialDesiredCount which explicitly sets the DesiredCount when the Service gets created.

Which service(s) is this request for?
Fargate, ECS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
The problem comes in combination with CloudFormation. As soon as anything changes in the ECS Service Resource (e.g. tags are modified), CloudFormation overwrites the DesiredCount. But DesiredCount might have changed because of AutoScaling. This can lead to situations where the number of running tasks is suddenly scaled down from e.g. 100 to 2.

Are you currently working around this issue?
There's no real workaround , as Cfn does not provide a conditional value based when the stack gets created or updated.

The only thing you can do is to remove the DesiredCount property from the CloudFormation template after the application gets the first time deployed.

Attachments
Initially opened the ticket at the CloudFormation roadmap here, but got the hint that it's actually an ECS issue.

[jooj123]

We have added a hack around this to do aws cli call to get current desiredCount and pass that to CF script (if it exists)

Feels very clunky / error prone and something that should be handled by AWS natively - would love for this to be sorted please 🙏

[softprops]

We do something similar passing in a desired count and the number of active services which will be one or zero. so that we leave autoscaling's desired count be when a service has been previously created and to set an initial value if its the first time the service is being created

<snip>
Parameters:
  DesiredCount:
    Type: Number
    Description: default number of tasks for a service when a service does not yet exist
  ServiceCount:
    Type: Number
    Description: precise number services with the same name in the same cluster

Conditions:
  ServiceExists:
    !Equals [!Ref ServiceCount, 1]

Resources:
  Service:
    Type: AWS::ECS::Service
    Properties:
      DesiredCount: !If [ServiceExists, !Ref 'AWS::NoValue', !Ref 'DesiredCount']
<snip>

the pass the output of the following to our cloud formation deploy command as a parameter override for ServiceCount

aws ecs describe-services \
  --cluster {the-cluster}  \
  --services {the-service} \
  --query 'length(services[?status==`ACTIVE`])' --output text

[surfkansas]

Our "hack" involves telling CFN whether this is an initial rollout and then only specifying DesiredCount for initial rollouts.

{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Parameters": {
        "CleanRollout": {
            "Description": "If set to 'Yes', this will do a clean service creation with all desired counts set to zero",
            "Default": "No",
            "Type": "String",
            "AllowedValues": [ "No", "Yes" ]
        }
    },
    "Conditions": {
        "IsCleanRollout": { "Fn::Equals": [ { "Ref": "CleanRollout" }, "Yes" ] }
    },
    "Resources": {
        "MySweetService": {
            "Type": "AWS::ECS::Service",
            "Properties": {
                "TaskDefinition": "my-amazing-task:3",
                "ServiceName": "my-sweet-service",
                "LaunchType": "FARGATE",
                "DeploymentConfiguration": {
                    "MaximumPercent": 200,
                    "MinimumHealthyPercent": 100
                },
                "DesiredCount": { 
                    "Fn::If": [ "IsCleanRollout", 0, { "Ref": "AWS::NoValue" } ] 
                },
                "Cluster": "my-perfect-cluster"
            }
        }
    }
}

I would very much love the ability to simplify this down and not have to tell CFN if we are doing an initial rollout.

[jooj123]

Is there any plan for this one @pavneeta ?

[ghost]

REA Group tried to implement an initialDesiredCount-like behaviour, based on the knowledge that CloudFormation only requires DesiredCount when the ECS Service resource is created. We have learnt that this is not a good approach.

When we submit our CloudFormation stack change, we need to be able to determine if DesiredCount is required, or not. Here are the factors we need to consider:

1. Does the ECS Service resource exist yet? If it doesn't exist yet, DesiredCount is mandatory
2. Will the ECS Service resource require replacement? In the case of replacement, DesiredCount is mandatory.

It seemed feasible to test for 1, but it's not trivial to succeed at 2 in a maintainable way.

At this stage, the only feasible workaround seems to be to always set DesiredCount when we update the stack. I hope you can find a good way for customers to safely deploy their ECS Services using CloudFormation.

[itaykat]

Is this implemented in latest CDK?

[itaykat]

Our "hack" involves telling CFN whether this is an initial rollout and then only specifying DesiredCount for initial rollouts.

{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Parameters": {
        "CleanRollout": {
            "Description": "If set to 'Yes', this will do a clean service creation with all desired counts set to zero",
            "Default": "No",
            "Type": "String",
            "AllowedValues": [ "No", "Yes" ]
        }
    },
    "Conditions": {
        "IsCleanRollout": { "Fn::Equals": [ { "Ref": "CleanRollout" }, "Yes" ] }
    },
    "Resources": {
        "MySweetService": {
            "Type": "AWS::ECS::Service",
            "Properties": {
                "TaskDefinition": "my-amazing-task:3",
                "ServiceName": "my-sweet-service",
                "LaunchType": "FARGATE",
                "DeploymentConfiguration": {
                    "MaximumPercent": 200,
                    "MinimumHealthyPercent": 100
                },
                "DesiredCount": { 
                    "Fn::If": [ "IsCleanRollout", 0, { "Ref": "AWS::NoValue" } ] 
                },
                "Cluster": "my-perfect-cluster"
            }
        }
    }
}

I would very much love the ability to simplify this down and not have to tell CFN if we are doing an initial rollout.

Unfortunately core.Aws.NO_VALUE one is impossible to assign for desired_count while using CDK... any workarounds here?

[guoqings]

We are making following improvements to ECS integration with Cloudformation:

1. DesiredCount becomes an optional field in CFN CreateService and DesiredCount=1 would be used as the default value if it is missing
2. UpdateService will also omit DesiredCount when it is missing in CFN template

Customers expect the current behavior (i.e. UpdateService will use the set DesiredCount in the CFN template) can just add DesiredCount in their CFN templates. Existing customers wanting the new behavior can get it by removing DesiredCount from their CFN templates. The changes will be released soon.

[guoqings]

This improvement has been released to all the regions.

[pgarbe]

Is this somewhere documented?