env deploy after CLI update

[benjaminpottier]

We're in the process of moving to using the new environments manifest. We moved from CLI version 1.17.0 to 1.21.0. I generated the manifest for our dev environment per instructions from the blog post and then ran copilot env deploy --name dev.

However, there were a number of failures during the update that we're not sure how to move past:

- Creating the infrastructure for the insuredportal-dev environment.                 [update rollback failed]  [102.3s]
  Export insuredportal-dev-ServiceDiscoveryNamespaceID cannot be updated
   as it is in use by insuredportal-dev-graphql, insuredportal-dev-ivr-s
  erver and insuredportal-dev-spa (and 1 more)
  The following resource(s) failed to update: [Cluster, DNSDelegationFun
  ction, CertificateValidationFunction, EnvironmentManagerRole].

  - An ECS cluster to group your services                                            [update failed]           [2.6s]
    Resource handler returned message: "Error occurred during operation 'S
    ettings can only be modified, not removed. Required Settings: [contain
    erInsights]'." (RequestToken: d3ec517c-99fd-a696-be4f-1d1a32c57532, Ha
    ndlerErrorCode: GeneralServiceException)

  - An IAM Role to describe resources in your environment                            [update failed]           [20.9s]
    Resource update cancelled

[efekarakus]

Hi @benjaminpottier!

Oh no 🙇 that is super strange, each environment should be getting its own service discovery namespace 🤔

Can you tell me a little bit more about the setup:

1. Is thedev environment importing a VPC?
2. Would you mind sharing the environment manifest file?
3. When you run copilot env package do you see a difference in the template generated compared to what's stored in CloudFormation? (I use a tool like https://www.yamldiff.com/ to highlight the differences between the two templates)
4. Were there any modifications done to the resources such as the service discovery namespace outside of Copilot? through the AWS CLI or Console for example

[benjaminpottier]

Hi @benjaminpottier!

Oh no 🙇 that is super strange, each environment should be getting its own service discovery namespace 🤔

Can you tell me a little bit more about the setup:

1. Is thedev environment importing a VPC?
2. Would you mind sharing the environment manifest file?
3. When you run copilot env package do you see a difference in the template generated compared to what's stored in CloudFormation? (I use a tool like https://www.yamldiff.com/ to highlight the differences between the two templates)
4. Were there any modifications done to the resources such as the service discovery namespace outside of Copilot? through the AWS CLI or Console for example

1. All our environments used the VPC created by copilot.
2. All that is in the manifest file is "name:dev" and "type: Envrionment". I generated it from env show command.
3. YAML diff is complaining about the format from the env package output. See:

Error in left input: unknown tag !<!Ref> at line 46, column 38:
    ... Not [!Equals [ !Ref ALBWorkloads, "" ]]

4. We have made modifications outside copilot, but not to the service discovery. It might be worth noting that our service discovery namespaces never had the environment included in them before, except for our prod environment. We always thought this was strange. What I mean is, in our dev, test, and model environments we have <app>.local and in prod we have <env>.<app>.local.

[efekarakus]

Ohh!! got it!!

Can you try the release with v1.21.1 instead?

We had discovered a bug in our translation of the manifest, here is the snippet from the release notes:

Preserve existing service discovery endpoint (#3949)

In the transition from env upgrade to env deploy, we lost the preservation of the ServiceDiscoveryEndpoint parameter and instead assumed the [app].[env].local format. However, environments that predated our v1.9.0 release have [app].local-formatted ServiceDiscoveryEndpoint parameters, and therefore were erroring out when updates were attempted. This fix preserves the existing value when env deploy is run.

[benjaminpottier]

Ohh!! got it!!

Can you try the release with v1.21.1 instead?

We had discovered a bug in our translation of the manifest, here is the snippet from the release notes:

Preserve existing service discovery endpoint (#3949)

In the transition from env upgrade to env deploy, we lost the preservation of the ServiceDiscoveryEndpoint parameter and instead assumed the [app].[env].local format. However, environments that predated our v1.9.0 release have [app].local-formatted ServiceDiscoveryEndpoint parameters, and therefore were erroring out when updates were attempted. This fix preserves the existing value when env deploy is run.

I would but am unable because the stack in a "UPDATE_ROLLBACK_FAILED state and can not be updated."

I have the option to "Continue update rollback" but am hesitant because of the following warning:

After the rollback is complete, the state of the skipped resources will be inconsistent with the state of the resources in the stack template. Before performing another stack update, you must update the stack or resources to be consistent with each other. If you don't, subsequent stack updates might fail, and the stack will become unrecoverable.

What would you suggest?

[efekarakus]

Gotcha, this scenario sounds very similar to this thread: https://gitter.im/aws/copilot-cli?at=6306516cb16e8236e3287045

In this situation, a client continued the rollback skipping the Cluster and EnvironmentManagerRole which got them to UPDATE_ROLLBACK_COMPLETE afterwards it looks like with v1.21.1 the command could succeed again.

[benjaminpottier]

That kind of work. But now I'm getting:

Statement IDs (SID) in a single policy must be unique. (Service: AmazonIdentityManagement; Status Code: 400; Error Code: MalformedPolicyDocument; Request ID: 4c4def8e-cca8-4042-884a-850a1a1bcd7a; Proxy: null)

The following resource(s) failed to update: [EnvironmentManagerRole].

Also, when I run env package now I get the following:

Only found one environment, defaulting to: dev
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x12ddede]

goroutine 1 [running]:
github.com/aws/copilot-cli/internal/pkg/describe.NewEnvDescriber({{0xc0004c8310, 0xd}, {0xc000694748, 0x3}, 0x0, {0x0, 0x0}, {0x0, 0x0}})
        /codebuild/output/src864945534/src/internal/pkg/describe/env.go:78 +0x7e
github.com/aws/copilot-cli/internal/pkg/cli/deploy.NewEnvDeployer(0xc0008d4c90)
        /codebuild/output/src864945534/src/internal/pkg/cli/deploy/env.go:112 +0x2cd
github.com/aws/copilot-cli/internal/pkg/cli.newPackageEnvOpts.func2()
        /codebuild/output/src864945534/src/internal/pkg/cli/env_package.go:112 +0xbb
github.com/aws/copilot-cli/internal/pkg/cli.(*packageEnvOpts).Execute(0xc000119ad0)
        /codebuild/output/src864945534/src/internal/pkg/cli/env_package.go:153 +0x24d
github.com/aws/copilot-cli/internal/pkg/cli.run({0x1f0cd70, 0xc000119ad0})
        /codebuild/output/src864945534/src/internal/pkg/cli/cli.go:98 +0x59
github.com/aws/copilot-cli/internal/pkg/cli.buildEnvPkgCmd.func1(0x0?, {0x0?, 0x0?, 0x0?})
        /codebuild/output/src864945534/src/internal/pkg/cli/env_package.go:282 +0x65
github.com/aws/copilot-cli/internal/pkg/cli.runCmdE.func1(0xc000426000?, {0x2caa6f0?, 0x0?, 0x0?})
        /codebuild/output/src864945534/src/internal/pkg/cli/cli.go:72 +0x7b
github.com/spf13/cobra.(*Command).execute(0xc000426000, {0x2caa6f0, 0x0, 0x0})
        /go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:872 +0x694
github.com/spf13/cobra.(*Command).ExecuteC(0xc0003d1400)
        /go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:990 +0x3b4
github.com/spf13/cobra.(*Command).Execute(...)
        /go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:918
main.main()
        /codebuild/output/src864945534/src/cmd/copilot/main.go:34 +0x25

This is with running on macOS M1 and Linux x64, version 1.22.0.

[benjaminpottier]

That kind of work. But now I'm getting:

Statement IDs (SID) in a single policy must be unique. (Service: AmazonIdentityManagement; Status Code: 400; Error Code: MalformedPolicyDocument; Request ID: 4c4def8e-cca8-4042-884a-850a1a1bcd7a; Proxy: null)

The following resource(s) failed to update: [EnvironmentManagerRole].

Also, when I run env package now I get the following:

Only found one environment, defaulting to: dev
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x12ddede]

goroutine 1 [running]:
github.com/aws/copilot-cli/internal/pkg/describe.NewEnvDescriber({{0xc0004c8310, 0xd}, {0xc000694748, 0x3}, 0x0, {0x0, 0x0}, {0x0, 0x0}})
        /codebuild/output/src864945534/src/internal/pkg/describe/env.go:78 +0x7e
github.com/aws/copilot-cli/internal/pkg/cli/deploy.NewEnvDeployer(0xc0008d4c90)
        /codebuild/output/src864945534/src/internal/pkg/cli/deploy/env.go:112 +0x2cd
github.com/aws/copilot-cli/internal/pkg/cli.newPackageEnvOpts.func2()
        /codebuild/output/src864945534/src/internal/pkg/cli/env_package.go:112 +0xbb
github.com/aws/copilot-cli/internal/pkg/cli.(*packageEnvOpts).Execute(0xc000119ad0)
        /codebuild/output/src864945534/src/internal/pkg/cli/env_package.go:153 +0x24d
github.com/aws/copilot-cli/internal/pkg/cli.run({0x1f0cd70, 0xc000119ad0})
        /codebuild/output/src864945534/src/internal/pkg/cli/cli.go:98 +0x59
github.com/aws/copilot-cli/internal/pkg/cli.buildEnvPkgCmd.func1(0x0?, {0x0?, 0x0?, 0x0?})
        /codebuild/output/src864945534/src/internal/pkg/cli/env_package.go:282 +0x65
github.com/aws/copilot-cli/internal/pkg/cli.runCmdE.func1(0xc000426000?, {0x2caa6f0?, 0x0?, 0x0?})
        /codebuild/output/src864945534/src/internal/pkg/cli/cli.go:72 +0x7b
github.com/spf13/cobra.(*Command).execute(0xc000426000, {0x2caa6f0, 0x0, 0x0})
        /go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:872 +0x694
github.com/spf13/cobra.(*Command).ExecuteC(0xc0003d1400)
        /go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:990 +0x3b4
github.com/spf13/cobra.(*Command).Execute(...)
        /go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:918
main.main()
        /codebuild/output/src864945534/src/cmd/copilot/main.go:34 +0x25

This is with running on macOS M1 and Linux x64, version 1.22.0.

I was able to get the package command to start working by downgrading to 1.21.1. I can see from cloudtrail that its trying to update the root policy for dev EnvManagerRole with a policy that does have two of the same Sids, "PatchPutObjectsToArtifactBucket".

[benjaminpottier]

Sorry, but I'm really hoping to work through this issue today so I can have a solution to move forward with the rest of my environments.

I fixed the EnvManagerRole issue, but now when I try to deploy the environment I get:

Export insuredportal-dev-SubDomain cannot be deleted as it is in use by insuredportal-dev-graphql, insuredportal-dev-ivr-server and insuredportal-dev-spa (and 1 more)

Also, I'm really worried this is in a state where I have to delete and re-create my dev environment which would be extremely disruptive. It also doesn't give me a lot of confidence in moving forward with the rest of my environments. Any help is appreciated.

[efekarakus]

I fixed the EnvManagerRole issue

Awesome!

Export insuredportal-dev-SubDomain cannot be deleted

Hmm, is there a domain that's associated with the application? was the application created with copilot app init --domain? That error means that for some reason copilot couldn't find a domain name associated with the application but the stack seems to think there should be one.

To troubleshoot, in the application account if you go to AWS Systems Manager > Parameter Store do you see your application have a domain name value or is it an empty string?

[benjaminpottier]

I fixed the EnvManagerRole issue

Awesome!

Export insuredportal-dev-SubDomain cannot be deleted

Hmm, is there a domain that's associated with the application? was the application created with copilot app init --domain? That error means that for some reason copilot couldn't find a domain name associated with the application but the stack seems to think there should be one.

To troubleshoot, in the application account if you go to AWS Systems Manager > Parameter Store do you see your application have a domain name value or is it an empty string?

I do see the domain value. Could it be when I "Skipped" those resources to fix the failed rollback that might have broken something? I'm also noticing things missing like S3Bucket for copilot specific functions when I run a env package -n dev. The stack seems really out of whack.

[efekarakus]

💭 I'm trying to think about what could be the issue here.

I'm confused because if there is a domain value, then the insuredportal-dev-SubDomain shouldn't be getting deleted.
When you run copilot env package what are the lines that get deleted? Do you see an EnvironmentSubdomain output written there.

[benjaminpottier]

S3Key and S3Bucket are empty for CertificateValidationFunction, CustomDomainFunction, and DNSDelegationFunction.

This is what I'm seeing for EnvironmentSubdomain:

  EnvironmentSubdomain:
    Condition: DelegateDNS
    Value: !Sub ${EnvironmentName}.${AppName}.${AppDNSName}
    Description: The domain name of this environment.
    Export:
      Name: !Sub ${AWS::StackName}-SubDomain

[efekarakus]

ah the empty values should be okay, if you use copilot env package --upload-assets those values will be filled.

Would you mind running copilot env deploy again and giving a screenshot of the Events tab in CloudFormation for the resources?

[benjaminpottier]

ah the empty values should be okay, if you use copilot env package --upload-assets those values will be filled.

Would you mind running copilot env deploy again and giving a screenshot of the Events tab in CloudFormation for the resources?

I'm hitting the policy issue again. I have to update the stack directly and remove the PatchPutObjectsToArtifactBucket statement to get passed the error. Is there a better way?

[efekarakus]

Gotcha, in the CloudFormation template right now what do you see for the environment's version?

Description: CloudFormation environment template for infrastructure shared among Copilot workloads.
Metadata:
  Version: v1.12.2 # What is this value for you?

I think for now if you update the IAM role directly in the console by giving it a different sidfor PatchPutObjectsToArtifactBucket that should unblock the stack (like PatchPutObjectsToArtifactBucket-backup)