Pipeline for service fails if service is not already deployed to environment.

[CorinWilkins]

We have a pipeline running in a build account, it deploys to an environment in another account. If the service hasn't already been deployed to an account the cloudformation stage will fail with some variation of a S3 permissions error.

Resource handler returned message: "Your access has been denied by S3, please make sure your request credentials have permission to GetObject for stackset-app-infrastruc-pipelinebuiltartifactbuc-xyz/manual/scripts/custom-resources/envcontrollerfunction/3ffcf03598029891816b7ce2d1ff14fdd8079af4406a0cfeff1d4aa0109dcd7d.zip. S3 Error Code: AccessDenied. S3 Error Message: Access Denied (Service: Lambda, Status Code: 403, Request ID: aae1a382-3a20-4345-80ca-ac160bc75175)" (RequestToken: ccbd59fc-caa1-dd84-e5a9-9b5e871070bb, HandlerErrorCode: AccessDenied)

We are potentially going to have many apps, and many environments so at best this will be very inconvenient. Worse we restrict access to our prod account so manual deploys are not be possible.

[Lou1415926]

Hello @CorinWilkins ! Thank you for reporting this issue.

I have a few vanilla questions to start with while I'm attempting to reproduce the issue:

1. Which copilot version are you running on your local machine? You can check by running copilot version.
2. Which copilot version are you running on your pipeline? You can check by looking for a line similar to - wget -q https://ecs-cli-v2-release.s3.amazonaws.com/copilot-linux-v1.21.1in the buildspec.yml file for the pipeline.
3. Just to confirm, this S3 error only happens for services that have not deployed to the environment; it doesn't happen to any services that are already in the environment right?
4. You mentioned that the env is in a different account. Do you happen to know if the same error happen if the environment that is in the same account of the pipeline?

Thank you very much and sorry for the inconvenience!

Edit:
I haven't been able to reproduce the issue. Here is what I did:

1. I have an environment called vanilla, and another called cross-account. vanilla is deployed in the same account as the pipeline, while cross-account is in a different account. Both environments are on version v1.12.2.
2. I have two services, web and backend. They haven't been deployed to either of the environments.
3. copilot pipeline init, stages are vanilla and cross-account.
4. Push the codes to repo, and run copilot pipeline deploy
5. The pipeline was able to succeed through every stage.

Any information on your setup would be very helpful for me to reproduce the issue. Thank you!

[CorinWilkins]

1. We are running copilot v1.21.1 locally
2. Our pipelines also use copilot v1.21.1
3. We only get the S3 permissions issue when a service has not already been deployed to an environment. Nb we have seen this issue for several services but it isn't always the same resource in S3.
4. We do not have an environment in the same account as the pipelines.

More detail on our setup:
We have a build account where the app is deployed. We use the build account to run all copilot commands. We have a dev account to which we deployed the dev environment from the build account by specifying the dev account profile in copilot env init. Our pipelines are deployed to the build account.

[iamhopaul123]

Hello @CorinWilkins. Could you try to run env deploy for all the envs you have so far, to apply the patch to the env manager role to make sure they have proper permission associated and then trigger the pipeline again to see if it works (make sure both your local CLI and the one used by the pipeline buildspec are up-to-date)? Thank you!

[CorinWilkins]

I have redeployed each environment into its respective accounts. I've also added an environment to our build account and I'm still getting the permissions denied error.

The app is v1.0.2
The app, environments and services were all created with copilot v1.21.1

[iamhopaul123]

This is strange. This looks very similar to #3453 (comment) which we resolved before. Could you help us to check the permission for the files that failed to get? Also, which resource creation returns this error msg? Thank you!

[CorinWilkins]

Resource handler returned message: "Your access has been denied by S3, please make sure your request credentials have permission to GetObject for stackset-...-infrastruc-pipelinebuiltartifactbuc-sic0fgzbzvkl/manual/scripts/custom-resources/rulepriorityfunction/584fba94a6e8a3ba8893428c31648e101fb8bf840647ce0b4505ffc86088ab50.zip. S3 Error Code: AccessDenied. S3 Error Message: Access Denied (Service: Lambda, Status Code: 403, Request ID: cb919aa9-9bec-4890-8a27-86619824f4a8)" (RequestToken: 5c3055ae-97d4-d5b9-ff33-760882c85b14, HandlerErrorCode: AccessDenied)

To replicate:

1. I created a loadbalanced service called "foo". Running an existing image(nginx) on port 80
2. I then redeployed the pipeline and committed and pushed my changes.

Looking at the permissions for the object above in s3 the "Object owner (external account)" is our production account. We have 4 accounts. Build, Sandbox, Dev, and Prod. The pipeline is failing to deploy to sandbox, the first stage in our pipeline.
There is also a permission for the build account to read/write which is the account the bucket lives in.

[CorinWilkins]

I've manually added the sandbox account to the grantees and re-ran the pipeline but this has not worked.

*Edit: It looks like running the pipeline removes the grantee I added manually.

[iamhopaul123]

Hello @CorinWilkins. I think we have clues on the root of the issue and I'm trying to reproduce it. Once we figured out the root cause I'll send out a quick fix which is expected to be included in our next release!

[efekarakus]

This fix is now released in v1.22 🚀 !

For the blog post: https://aws.github.io/copilot-cli/blogs/release-v122/
Release notes: https://github.com/aws/copilot-cli/releases/tag/v1.22.0

[efekarakus]

Once v1.22 is downloaded, to fix the issue you'd need to run copilot app upgrade to update the template of the S3 buckets to set object ownership to bucket owner. Hope this helps!