Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pipeline for service fails if service is not already deployed to environment. #3984

Closed
CorinWilkins opened this issue Sep 8, 2022 · 10 comments · Fixed by #3998
Closed

Pipeline for service fails if service is not already deployed to environment. #3984

CorinWilkins opened this issue Sep 8, 2022 · 10 comments · Fixed by #3998

Comments

@CorinWilkins
Copy link

We have a pipeline running in a build account, it deploys to an environment in another account. If the service hasn't already been deployed to an account the cloudformation stage will fail with some variation of a S3 permissions error.

Resource handler returned message: "Your access has been denied by S3, please make sure your request credentials have permission to GetObject for stackset-app-infrastruc-pipelinebuiltartifactbuc-xyz/manual/scripts/custom-resources/envcontrollerfunction/3ffcf03598029891816b7ce2d1ff14fdd8079af4406a0cfeff1d4aa0109dcd7d.zip. S3 Error Code: AccessDenied. S3 Error Message: Access Denied (Service: Lambda, Status Code: 403, Request ID: aae1a382-3a20-4345-80ca-ac160bc75175)" (RequestToken: ccbd59fc-caa1-dd84-e5a9-9b5e871070bb, HandlerErrorCode: AccessDenied)

We are potentially going to have many apps, and many environments so at best this will be very inconvenient. Worse we restrict access to our prod account so manual deploys are not be possible.

@Lou1415926
Copy link
Contributor

Lou1415926 commented Sep 8, 2022

Hello @CorinWilkins ! Thank you for reporting this issue.

I have a few vanilla questions to start with while I'm attempting to reproduce the issue:

  1. Which copilot version are you running on your local machine? You can check by running copilot version.
  2. Which copilot version are you running on your pipeline? You can check by looking for a line similar to - wget -q https://ecs-cli-v2-release.s3.amazonaws.com/copilot-linux-v1.21.1in the buildspec.yml file for the pipeline.
  3. Just to confirm, this S3 error only happens for services that have not deployed to the environment; it doesn't happen to any services that are already in the environment right?
  4. You mentioned that the env is in a different account. Do you happen to know if the same error happen if the environment that is in the same account of the pipeline?

Thank you very much and sorry for the inconvenience!

Edit:
I haven't been able to reproduce the issue. Here is what I did:

  1. I have an environment called vanilla, and another called cross-account. vanilla is deployed in the same account as the pipeline, while cross-account is in a different account. Both environments are on version v1.12.2.
  2. I have two services, web and backend. They haven't been deployed to either of the environments.
  3. copilot pipeline init, stages are vanilla and cross-account.
  4. Push the codes to repo, and run copilot pipeline deploy
  5. The pipeline was able to succeed through every stage.

Any information on your setup would be very helpful for me to reproduce the issue. Thank you!

@CorinWilkins
Copy link
Author

CorinWilkins commented Sep 9, 2022

  1. We are running copilot v1.21.1 locally
  2. Our pipelines also use copilot v1.21.1
  3. We only get the S3 permissions issue when a service has not already been deployed to an environment. Nb we have seen this issue for several services but it isn't always the same resource in S3.
  4. We do not have an environment in the same account as the pipelines.

More detail on our setup:
We have a build account where the app is deployed. We use the build account to run all copilot commands. We have a dev account to which we deployed the dev environment from the build account by specifying the dev account profile in copilot env init. Our pipelines are deployed to the build account.

@iamhopaul123
Copy link
Contributor

iamhopaul123 commented Sep 12, 2022

Hello @CorinWilkins. Could you try to run env deploy for all the envs you have so far, to apply the patch to the env manager role to make sure they have proper permission associated and then trigger the pipeline again to see if it works (make sure both your local CLI and the one used by the pipeline buildspec are up-to-date)? Thank you!

@CorinWilkins
Copy link
Author

I have redeployed each environment into its respective accounts. I've also added an environment to our build account and I'm still getting the permissions denied error.

The app is v1.0.2
The app, environments and services were all created with copilot v1.21.1

@iamhopaul123
Copy link
Contributor

This is strange. This looks very similar to #3453 (comment) which we resolved before. Could you help us to check the permission for the files that failed to get? Also, which resource creation returns this error msg? Thank you!

@CorinWilkins
Copy link
Author

Resource handler returned message: "Your access has been denied by S3, please make sure your request credentials have permission to GetObject for stackset-...-infrastruc-pipelinebuiltartifactbuc-sic0fgzbzvkl/manual/scripts/custom-resources/rulepriorityfunction/584fba94a6e8a3ba8893428c31648e101fb8bf840647ce0b4505ffc86088ab50.zip. S3 Error Code: AccessDenied. S3 Error Message: Access Denied (Service: Lambda, Status Code: 403, Request ID: cb919aa9-9bec-4890-8a27-86619824f4a8)" (RequestToken: 5c3055ae-97d4-d5b9-ff33-760882c85b14, HandlerErrorCode: AccessDenied)

To replicate:

  1. I created a loadbalanced service called "foo". Running an existing image(nginx) on port 80
  2. I then redeployed the pipeline and committed and pushed my changes.

Looking at the permissions for the object above in s3 the "Object owner (external account)" is our production account. We have 4 accounts. Build, Sandbox, Dev, and Prod. The pipeline is failing to deploy to sandbox, the first stage in our pipeline.
There is also a permission for the build account to read/write which is the account the bucket lives in.

@CorinWilkins
Copy link
Author

CorinWilkins commented Sep 13, 2022

I've manually added the sandbox account to the grantees and re-ran the pipeline but this has not worked.

*Edit: It looks like running the pipeline removes the grantee I added manually.

@iamhopaul123
Copy link
Contributor

Hello @CorinWilkins. I think we have clues on the root of the issue and I'm trying to reproduce it. Once we figured out the root cause I'll send out a quick fix which is expected to be included in our next release!

@mergify mergify bot closed this as completed in #3998 Sep 14, 2022
mergify bot pushed a commit that referenced this issue Sep 14, 2022
Fixes #3984


By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the Apache 2.0 License.
@efekarakus
Copy link
Contributor

This fix is now released in v1.22 🚀 !

For the blog post: https://aws.github.io/copilot-cli/blogs/release-v122/
Release notes: https://github.com/aws/copilot-cli/releases/tag/v1.22.0

@efekarakus
Copy link
Contributor

Once v1.22 is downloaded, to fix the issue you'd need to run copilot app upgrade to update the template of the S3 buckets to set object ownership to bucket owner. Hope this helps!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants