unable to upgrade a Backend Service from 1.19 to 1.21

[codekitchen]

We have one backend service that we are trying to upgrade from copilot 1.19 to copilot 1.21 and we consistently get an error during deploy. I've already used env show --manifest to create the env manifest, and env deploy to make sure all the env updates are deployed. But svc deploy fails in cloudformation with:

✔ Proposing infrastructure changes for stack xxx-xxx-xxx
- Updating the infrastructure for stack xxx-xxx-xxx               [update rollback complete]  [60.5s]
  The following resource(s) failed to create: [LoadBalancerInternalDNSAl                                               
  ias]. The following resource(s) failed to update: [ExecutionRole, HTTP                                               
  RulePriorityAction].                                                                                                 
  - Update your environment's shared resources                                             [not started]                
  - An IAM Role for the Fargate agent to make AWS API calls on your behalf                 [update complete]           [21.7s]
    Resource update cancelled                                                                                          
  - A HTTP listener rule for forwarding HTTP traffic                                       [not started]                
  - A custom resource assigning priority for HTTP listener rules                           [update complete]           [1.7s]
    Resource update cancelled                                                                                          
  - Alias for xxx.xxx.xxx.internal to the internal load balancer  [delete complete]           [0.0s]
    CustomResource attribute error: Vendor response doesn't contain Intern                                             
    alWorkloadsHostedZone key in object arn:aws:cloudformation:us-west-2:0                                             
    xxx:stack/xxx-xxx-xxx/a5780ba0-f7d7-1                                             
    1ec-af7f-069360ba28a1|EnvControllerAction|744b47e9-ae0a-427a-904c-1f43                                             
    f0cd494e in S3 bucket cloudformation-custom-resource-storage-uswest2                                               
  - An ECS service to run and maintain your tasks in the environment cluster               [not started]                
  - An ECS task definition to group your containers and run them on ECS                    [not started]                
  - An IAM role to control permissions for the containers in your tasks                    [not started]                
✘ deploy service xxx to environment xxx: deploy service: stack xxx-xxx-xxx did not complete successfully and exited with status UPDATE_ROLLBACK_COMPLETE

This seems possibly related to defining a http alias? Which we aren't doing in this service. The relevant parts of the manifest file are:

name: xxx
type: Backend Service

http:
  path: '/'
  healthcheck:
    path: '/healthcheck'
    interval: 15s
    timeout: 5s
    grace_period: 15s

network:
  vpc:
    placement: private

I tried upgrading to copilot 1.20 first, but that version also gives the same error during deploy.

The generated environment manifest only has network configuration:

name: xxx
type: Environment
network:
    vpc:
        id: vpc-xxx
        subnets:
            public:
                - id: subnet-xxx
                - id: subnet-xxx
                - id: subnet-xxx
            private:
                - id: subnet-xxx
                - id: subnet-xxx
                - id: subnet-xxx

[huanjani]

Hello, @codekitchen!
Just checking: were your env and svc deployed with v1.19? And the internal ALB created successfully? It's odd that there's no http field in your generated env manifest. 🤔

[codekitchen]

Hi @huanjani I can confirm that this svc was deployed with v1.19. The internal ALB was created successfully (named like xxx-Inter-TA2R6W41L3A1) and its target group is pointing to this service's task instances.

However, there's two services defined in this copilot app/environment, an external one and this internal one. I am finding some evidence that the last time we deployed this service, the other service was deployed using copilot v1.18.1 around the same time. If I understand correctly, that could have "downgraded" or messed with some of the shared environment resources right? So maybe relevant.

[huanjani]

Hmmm. When the internal ALB was created, was a pre-existing hosted zone used?
The public ALB and private ALB should be able to coexist in your env. The alias bit is related to the fact that we create one like http://{service name}.{env name}.{app name}.internal.

[codekitchen]

We did specify a --domain when doing the app init, but no we did not import a pre-existing hosted zone when creating this service. We just let copilot create its default .internal and .local hosted zones.

Possibly interesting: I tried adding the http section to the environment manifest. env deploy succeeded just fine but I still get exactly the same error output when trying to svc deploy. This is what I added:

http:
  private:
    subnets:
        - subnet-xxx
        - subnet-xxx
        - subnet-xxx
    security_groups:
        ingress:
            from_vpc: true

[huanjani]

Thank you for your patience, @codekitchen!

It appears we have a bug where the EnvController isn't being triggered, so env stack Outputs like InternalWorkloadsHostedZone, above, are missing. While we work on a fix, a workaround for you would be to trigger an execution of the EnvController lambda. Go into the CloudFormation console and manually modify your backend service's template, adding a dummy property to the EnvControllerAction's Properties field, like the last line here:

  EnvControllerAction:
    Metadata:
      'aws:copilot:description': "Update your environment's shared resources"
    Type: Custom::EnvControllerFunction
    Properties:
      ServiceToken: !GetAtt EnvControllerFunction.Arn
      Workload: !Ref WorkloadName
      EnvStack: !Sub '${AppName}-${EnvName}'
      Parameters: []
      SomethingNewHere: "1234"

Please let us know if that works for you!
Thanks!

[codekitchen]

Thank you @huanjani I can confirm that workaround got us successfully deploying again!

[Lou1415926]

Hello folks! We just released a patch version v1.21.1: https://github.com/aws/copilot-cli/releases/tag/v1.21.1.

The patch will trigger the EnvController automatically to fetch the latest Outputs, so that you don't need to perform the workaround that's mentioned here

Sorry for any inconvenience 🙇‍♂️