Pipeline failed after upgrade to 1.21 with "Your access has been denied by S3" error

[niros1]

Hi,
After upgrading to v1.21 my pipeline broke, and the CFN failed with error "Your access has been denied by S3 please make sure your request credentials have permission to GetObject for ...manual/scripts/custom-resources/envcontrollerfunction/3ffcf03598029891....zip
My platform includes several accounts root, dev, and prod. the pipeline run in root and deploy to the relevant environment, the error occurs on CFN running on dev env.

Steps taken:

1. upgrading to V1.21 from V1.19
2. Redeploy my pipeline and push my code
3. I tried to grant permissions to the bucket policy and to CFN role but with no luck.
4. I tried to create a new pipeline by copilot pipeline init - same error
5. Found this ticket Cloudformation addon templates fail with S3 error: Access Denied after updating to 1.16.0 #3453 but seems like it relevant to an older version.

10X In advance.

[huanjani]

Hi @niros1!

🤔 This doesn't have to do with permissions, but have all of the environments been upgraded-- have you generated env templates and run env deploy for them?

[niros1]

Hi,
I wasn't aware of that.
So, I tried to do so by running copilot env package -n dev --force --output-dir ./environments and get the following error ✘ read manifest for environment "dev": file /Users/niro/dev/bitbucket/labrain-be/copilot/environments/dev/manifest.yml does not exists.
My environments were created long ago with an early version of Copilot, I don't have any of those env files.
I tried to create an env pipeline as well and got the same error.

Can you guide me how to generate the env file?

Thanks.

UPDATE
I manage to generate the files by running copilot env show --manifest

[niros1]

After running copilot env deploy I get the following error.
I know there is a --force flag but does it means it will create the environment from scratch and recreate the cluster. it's a bit risky for production.

Only found one environment, defaulting to: dev
✔ Proposing infrastructure changes for the bv-labrain-be-dev environment.

• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [59.2s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [59.3s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [59.4s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [59.5s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [59.6s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [59.7s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [59.8s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [59.9s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [60.0s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [60.1s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [60.2s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [60.4s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [60.5s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [60.6s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [60.7s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [60.8s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [60.9s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [61.0s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [61.1s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [61.2s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [61.3s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [61.4s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [61.5s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [61.6s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [61.7s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [61.8s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [61.9s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [62.0s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [62.1s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [62.2s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [62.3s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [62.4s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [62.5s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [62.6s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [62.7s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [62.8s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [62.9s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [63.0s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [63.1s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [63.2s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [63.3s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [63.4s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [63.5s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [63.6s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [63.7s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [63.8s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [63.9s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [64.0s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [64.1s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [64.2s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [64.3s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [64.4s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [64.5s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [64.6s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [64.7s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [64.8s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [64.9s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [65.0s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [65.1s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [65.2s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [65.3s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [65.4s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [65.5s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [65.6s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [65.7s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [65.8s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [65.9s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [66.0s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [66.1s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [66.2s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [66.3s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [66.4s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [66.5s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [66.6s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [66.7s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [66.8s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [66.9s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [67.0s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [67.1s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [67.2s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [67.3s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [67.4s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [67.5s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [67.6s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [67.7s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [67.8s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [67.9s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [68.0s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [68.1s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [68.2s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [68.3s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [68.4s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [68.5s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [68.6s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [68.7s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [68.8s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [68.9s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [69.0s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [69.1s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [69.2s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [69.3s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [69.4s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [69.5s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [69.6s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [69.7s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [69.8s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [69.9s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [70.0s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [70.1s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [70.2s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [70.3s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [70.4s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [70.5s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [70.6s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [70.7s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [70.8s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [70.9s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [71.0s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [71.1s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback in progress] [71.2s]
• Creating the infrastructure for the bv-labrain-be-dev environment. [update rollback failed] [71.3s]
  Export bv-labrain-be-dev-ServiceDiscoveryNamespaceID cannot be updated
  as it is in use by bv-labrain-be-dev-bv-svc-cons-whatsapp, bv-labrain
  -be-dev-bv-svc-cons-whatsapp-consumer and bv-labrain-be-dev-bv-svc-lab
  rain-brain-factory (and 9 more)
  The following resource(s) failed to update: [Cluster, EnvironmentManag
  erRole].
  • An ECS cluster to group your services [update failed] [4.2s]
    Resource handler returned message: "Error occurred during operation 'S
    ettings can only be modified, not removed. Required Settings: [contain
    erInsights]'." (RequestToken: 1cb30d21-5b27-6441-06a9-b9c81a10597c, Ha
    ndlerErrorCode: GeneralServiceException)
  • An IAM Role to describe resources in your environment [update failed] [20.5s]
    Resource update cancelled
  • A security group for your load balancer allowing HTTP traffic [create complete] [6.9s]
  • An Application Load Balancer to distribute public traffic to your services [update complete] [0.0s]
  • A private DNS namespace for discovering services within the environment [update complete] [48.3s]
    ✘ deploy environment dev: stack bv-labrain-be-dev did not complete successfully and exited with status UPDATE_ROLLBACK_FAILED

[Lou1415926]

Hello! Would you please go to the CloudFormation console, go to the Events tab, and let us know what are the events there for that particular deployment? We are especially looking for events that happen before UPDATE_FAILED_ROLLBACK_IN_PROGRESS, because we'd like to know what are the resources that cause the rollback. However, inclusion of the events that happens after that would be potentially helpful as well!

[niros1]

Sure, here it is

2022-08-26 13:44:55 UTC+0300	bv-labrain-be-dev	UPDATE_ROLLBACK_FAILED	The following resource(s) failed to update: [Cluster, EnvironmentManagerRole].
2022-08-26 13:44:54 UTC+0300	EnvironmentManagerRole	UPDATE_FAILED	Resource update cancelled
2022-08-26 13:44:54 UTC+0300	Cluster	UPDATE_FAILED	Resource handler returned message: "Error occurred during operation 'Settings can only be modified, not removed. Required Settings: [containerInsights]'." (RequestToken: 1cb30d21-5b27-6441-06a9-b9c81a10597c, HandlerErrorCode: GeneralServiceException)
2022-08-26 13:44:53 UTC+0300	PublicLoadBalancer	UPDATE_COMPLETE	-
2022-08-26 13:44:53 UTC+0300	EnvironmentManagerRole	UPDATE_IN_PROGRESS	-
2022-08-26 13:44:53 UTC+0300	PublicLoadBalancer	UPDATE_IN_PROGRESS	-
2022-08-26 13:44:53 UTC+0300	Cluster	UPDATE_IN_PROGRESS	-
2022-08-26 13:44:52 UTC+0300	ServiceDiscoveryNamespace	UPDATE_COMPLETE	-
2022-08-26 13:44:43 UTC+0300	bv-labrain-be-dev	UPDATE_ROLLBACK_IN_PROGRESS	Export bv-labrain-be-dev-ServiceDiscoveryNamespaceID cannot be updated as it is in use by bv-labrain-be-dev-bv-svc-cons-whatsapp, bv-labrain-be-dev-bv-svc-cons-whatsapp-consumer and bv-labrain-be-dev-bv-svc-labrain-brain-factory (and 9 more)
2022-08-26 13:44:39 UTC+0300	ServiceDiscoveryNamespace	UPDATE_COMPLETE	-
2022-08-26 13:44:13 UTC+0300	EnvironmentManagerRole	UPDATE_COMPLETE	-
2022-08-26 13:43:59 UTC+0300	PublicLoadBalancer	UPDATE_COMPLETE	-
2022-08-26 13:43:58 UTC+0300	EnvironmentHTTPSecurityGroupIngressFromPublicALB	CREATE_COMPLETE	-
2022-08-26 13:43:58 UTC+0300	PublicLoadBalancer	UPDATE_IN_PROGRESS	-
2022-08-26 13:43:58 UTC+0300	EnvironmentHTTPSecurityGroupIngressFromPublicALB	CREATE_IN_PROGRESS	Resource creation Initiated
2022-08-26 13:43:58 UTC+0300	EnvironmentHTTPSecurityGroupIngressFromPublicALB	CREATE_IN_PROGRESS	-
2022-08-26 13:43:56 UTC+0300	PublicHTTPLoadBalancerSecurityGroup	CREATE_COMPLETE	-
2022-08-26 13:43:55 UTC+0300	PublicHTTPLoadBalancerSecurityGroup	CREATE_IN_PROGRESS	Resource creation Initiated
2022-08-26 13:43:54 UTC+0300	ServiceDiscoveryNamespace	UPDATE_IN_PROGRESS	Resource creation Initiated
2022-08-26 13:43:54 UTC+0300	EnvironmentManagerRole	UPDATE_IN_PROGRESS	-
2022-08-26 13:43:53 UTC+0300	ServiceDiscoveryNamespace	UPDATE_IN_PROGRESS	Requested update requires the creation of a new physical resource; hence creating one.
2022-08-26 13:43:52 UTC+0300	Cluster	UPDATE_COMPLETE	-
2022-08-26 13:43:51 UTC+0300	Cluster	UPDATE_IN_PROGRESS	-
2022-08-26 13:43:50 UTC+0300	PublicHTTPLoadBalancerSecurityGroup	CREATE_IN_PROGRESS	-
2022-08-26 13:43:44 UTC+0300	bv-labrain-be-dev	UPDATE_IN_PROGRESS	User Initiated

[Lou1415926]

I see! Thanks for the information! Is your original ServiceDiscoveryEndpoint parameter before this failed deployment <app>.local instead of <env>.<app>.local?

[niros1]

Sorry, but I didn't understand your question, can you elaborate?

[Lou1415926]

Sorry for the confusion! In the CloudFormation console, go to the Parameters tab. We'd like to know whether:

1. You have the ServiceDiscoveryEndpoint parameter there
2. If you do, is the value of ServiceDiscoveryEndpoint parameter bv-labrain-be.local or dev.bv-lebrain-be.local.

[niros1]

this is the parameter

ServiceDiscoveryEndpoint | bv-labrain-be.local

ALBWorkloads bv-svc-labrain-mlflow -
Aliases - -
AppDNSDelegationRole - -
AppDNSName - -
AppName bv-labrain-be -
CreateHTTPSListener false -
CreateInternalHTTPSListener false -
EFSWorkloads - -
EnvironmentName dev -
InternalALBWorkloads - -
NATWorkloads - -
ServiceDiscoveryEndpoint bv-labrain-be.local -
ToolsAccountPrincipalARN arn:aws:iam::XXXXXX:root -

[Lou1415926]

Thanks for confirming!

This is the same issue as the Gitter issue. In short, your environment is created before v1.9.0, which uses the app.local endpoint format. The env deploy command shipped in v1.20.0 did not take into account the compatibility with environment that has app.local as the endpoint, and try to use env.app.local for all environment. This triggers an update to the ServiceDiscoveryNamespace resource because it tries to update it from app.local to env.app.local - this update is not allowed by CFN, since ServiceDiscoveryNamespace is exported and imported by your service stack already.

The fix PR has been created #3949!

[niros1]

OK, 10X a lot.
What can I do till the PR will be resolved?

[Lou1415926]

First let's get your environment stack out of the UPDATE_ROLLBACK_FAILED state, you can follow the instruction here https://gitter.im/aws/copilot-cli?at=6306516cb16e8236e3287045. In general, you will need to let CFN know that it should stop trying to rollback Cluster, and instead just rollback other resources.

Make sure your stack is back in an UPDATE_ROLLBACK_COMPLETE state after these. If not, please let me know.