Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

I got a below issue when I run my aws code pipeline. #5

Open
jeetugswm opened this issue Oct 18, 2018 · 28 comments
Open

I got a below issue when I run my aws code pipeline. #5

jeetugswm opened this issue Oct 18, 2018 · 28 comments

Comments

@jeetugswm
Copy link

AccessDenied: Access Denied status code: 403, request id: D18C063FBD, host id: q8QSdgeMXd9GzqiHAn9aR7e+Qh5TNjYfRjbXxQP73FZbZWjFj78IXQ= for primary source and source version arn:aws:s3:::pre-reqartifactbucket-1btcbp1ce/Infra-Prov/SourceOutp/EEB5s

I have give full s3 access but still access issue & tried alot.
Please let me know the solution to avoid this.

Originally posted by @jeetugswm in #1 (comment)

@OfirYaron
Copy link

OfirYaron commented Nov 13, 2018

Hi @jeetugswm
I'm having the same issue, did you manage to solve it?

@davidkelley
Copy link

Has this issue been resolved? I'm encountering the same problem. It seems the ToolsAccount/TestAccount doesn't own the object being uploaded by the DevAccount IAM Role?

@floydding
Copy link

+1

@davidkelley
Copy link

@anshrma is it possible to get some feedback on this problem? Would love to implement something similar for our own account, but this seems to be a blocker.

@floydding
Copy link

Hi yall, I have figured out my problem. It might not be the same problem that you guys have encountered but I am here to post what my problem was.

So everything in S3 is encrypted at rest. When you don't specify how you want to encrypt them, objects in S3 will be encrypted by the default KMS key. And other accounts won't be able to get access to objects in the bucket because they don't have that KMS key for decryption. So to get around this issue, you need to create your own KMS key and use it to encrypt (let the CodeBuild to use this KMS Key you have created in this case). Then allow roles in other accounts to use this key by configure AssumeRole permissions. From what I see, most S3 access denial happens at not being able to decrypt objects. And this is specified here Troubleshoot S3 403 Access Denied - encrypted objects will also cause 403 Access Denied.

Hopefully this helps yall

-Floyd

@davidkelley
Copy link

Hi @floydding -- thanks for reaching out with a suggestion solution! Unfortunately, the problem is between the Pipeline grabbing the source code from the dev account and it being accessible in the tools account. I think this is due to an incorrect ACL setting of "private" on pipeline artifacts.

@floydding
Copy link

floydding commented Dec 6, 2018

@davidkelley I encountered that issue too. I believe that could also be related to KMS key issue. I think you might need to specify EncryptionKey in your ArtifactStore. Please see this documentation: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-codepipeline-pipeline-artifactstore.html. Hopefully this is where your issue was/is.

@lestephane
Copy link

My personal experience is that @floydding's suggestion to specify EncryptionKey in ArtifactStore is the solution, as long as you grant key usage permissions to the codebuildrole and the role used to access the codecommit repository. If you run through the instructions in the refarch blog and readme, it should just work (disclaimer: I never did, more on that below). Running the instructions out of sequence or adding the slightest variation could cause problems, as is typical in complex CloudFormation scenarios.

If you need to troubleshoot this further, you'd need to

  • Cause the error to happen
  • Go to the Cloudtrail dashboard
  • Click "View all events"
  • From the top, expand each event until you see one with Error code: AccessDenied (or similar). You may have to wait for 15 minutes. You may have to do this in the codebuild account and if that does not yield anything, look in the same Cloudtrail section for the other (codecommit) account.
  • Click View event
  • Inspect the JSON for a clearer error message
  • Maybe post it in this issue after removing any sensitive information

I lost a few hours troubleshooting this exact problem, on a cross-account setup I was creating from scratch, referring to this repository for the occasional copy and paste. When I got the same error as you, I was stuck, and eventually created a ticket. AWS Premium Support was not much help. Hopefully these troubleshooting steps will.

@Tamrael
Copy link

Tamrael commented Feb 8, 2019

@lestephane thank you so much. Your Cloudtrail comment saved me from the hell of aws non descriptive error messages and delivered me to the heavens of knowing what the hell is going on

@quarryman
Copy link

@lestephane Hello.
Could you specify what you mean 'do this in the codebuild account'
In my setup CodeBuild just assumes role and I didnt get the idea.
thanx

@lestephane
Copy link

@quarryman do this in the codebuild account means

  • Go to the AWS console of the account where the codebuild project is configured
  • Go to the Cloudtrail dashboard
  • Click "View all events"
  • etc...

@verma1997
Copy link

@jeetugswm It worked for me when I've given S3FullAccess permission to the codebuild role.

@meetreks
Copy link

Code build Service Account will not have access to the S3 artifacts created by Code Commit. So you need to find the service account role and add S3 full access or edit inline policy to add the S3 bucket with wildcards added so that this new bucket details get included and then it will work like magic.,

@sanjay-collab
Copy link

adding S3FullAccess permission to the codebuild role works

@tomarv2
Copy link

tomarv2 commented Jun 17, 2020

This is the policy that worked for me:

policy = <<-EOF
{
  "Version": "2012-10-17",
  "Statement": [
      {
          "Sid": "VisualEditor0",
          "Effect": "Allow",
          "Action": [
              "codebuild:CreateReportGroup",
              "codebuild:CreateReport",
              "logs:CreateLogStream",
              "codebuild:UpdateReport",
              "codebuild:BatchPutTestCases",
              "logs:PutLogEvents"
          ],
          "Resource": [
              "arn:aws:codebuild:us-east-2:123456789012:report-group/demo-ecs-pipeline-*",
              "arn:aws:logs:us-east-2:123456789012:log-group:/aws/codebuild/demo-ecs-pipeline",
              "arn:aws:logs:us-east-2:123456789012:log-group:/aws/codebuild/demo-ecs-pipeline:*"
          ]
      },
      {
          "Sid": "VisualEditor1",
          "Effect": "Allow",
          "Action": [
              "kms:EnableKey",
              "kms:GetPublicKey",
              "kms:ImportKeyMaterial",
              "kms:Decrypt",
              "kms:GenerateRandom",
              "kms:GenerateDataKeyWithoutPlaintext",
              "kms:Verify",
              "kms:ListResourceTags",
              "kms:CancelKeyDeletion",
              "ecr:UploadLayerPart",
              "kms:GenerateDataKeyPair",
              "kms:GetParametersForImport",
              "kms:DescribeCustomKeyStores",
              "kms:DeleteCustomKeyStore",
              "kms:UpdateCustomKeyStore",
              "kms:Encrypt",
              "kms:GetKeyRotationStatus",
              "kms:ScheduleKeyDeletion",
              "ecr:CompleteLayerUpload",
              "kms:ReEncryptTo",
              "kms:DescribeKey",
              "kms:CreateKey",
              "kms:ConnectCustomKeyStore",
              "kms:Sign",
              "ecr:BatchCheckLayerAvailability",
              "kms:EnableKeyRotation",
              "kms:ListKeyPolicies",
              "kms:UpdateKeyDescription",
              "kms:ListRetirableGrants",
              "kms:GetKeyPolicy",
              "ecr:GetAuthorizationToken",
              "kms:DeleteImportedKeyMaterial",
              "kms:GenerateDataKeyPairWithoutPlaintext",
              "kms:DisableKey",
              "kms:ReEncryptFrom",
              "kms:ListGrants",
              "kms:UpdateAlias",
              "kms:CreateCustomKeyStore",
              "ecr:PutImage",
              "kms:ListKeys",
              "kms:ListAliases",
              "kms:GenerateDataKey",
              "kms:CreateAlias",
              "kms:DisconnectCustomKeyStore",
              "ecr:InitiateLayerUpload",
              "kms:DeleteAlias"
          ],
          "Resource": "*"
      },
      {
          "Sid": "VisualEditor2",
          "Effect": "Allow",
          "Action": "logs:CreateLogGroup",
          "Resource": [
              "arn:aws:logs:us-east-2:123456789012:log-group:/aws/codebuild/demo-ecs-pipeline",
              "arn:aws:logs:us-east-2:123456789012:log-group:/aws/codebuild/demo-ecs-pipeline:*"
          ]
      },
      {
          "Sid": "VisualEditor3",
          "Effect": "Allow",
          "Action": "s3:*",
          "Resource": "arn:aws:s3:::codepipeline-us-east-2-*"
      }
  ]
}
EOF
}

@stoem
Copy link

stoem commented Aug 16, 2020

I wouldn't recommend giving full S3 access. I added these permissions to the CodeBuildBasePolicy-xxx policy, then edited them to specify the ARN (bucket name) that I wanted to access. That worked for me

s3:PutObject
s3:GetObject
s3:GetObjectVersion

If the ARN/bucket restriction is hard to figure out then give it access to all buckets, that's still safer than giving full S3 access as you are restricting it to 3 privileges.

@lestephane
Copy link

lestephane commented Aug 16, 2020 via email

aahung added a commit to aahung/codebuild-by-cloudformation that referenced this issue Nov 4, 2020
@sanjogj43
Copy link

@jeetugswm It worked for me when I've given S3FullAccess permission to the codebuild role.

Exactly what I needed. I was setting up the pipeline for learning purpose and, mostly this is what you need.

@Ochaun
Copy link

Ochaun commented Apr 2, 2021

@davidkelley I encountered that issue too. I believe that could also be related to KMS key issue. I think you might need to specify EncryptionKey in your ArtifactStore. Please see this documentation: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-codepipeline-pipeline-artifactstore.html. Hopefully this is where your issue was/is.

☝🏾 This works. @lestephane and @tomarv2 also made great additions. If you are running a pipeline with codebuild AND a custom KMS key. Make sure that the pipeline service role and the build service role have encrypt/decrypt access to the KMS key if you're using one. This is what I had to add to my KMS key policy.

{
              "Sid": "Allow use of the key",
              "Effect": "Allow",
              "Principal": {
                "AWS": [
                  {
                    "Fn::GetAtt": ["SomeBuildServiceRole", "Arn"]
                  },
                  {
                    "Fn::GetAtt": ["SomePipelineServiceRole", "Arn"]
                  }
                ]
              },
              "Action": [
                "kms:DescribeKey",
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey",
                "kms:GenerateDataKeyWithoutPlaintext"
              ],
              "Resource": "*"
            }

@JaghutTyrant
Copy link

Thanks @floydding, I think that in my case it really was an issue with the KMS key. I had created a S3 Bucket outside Codepipeline, and the source stage could in fact create the SourceArtifact but, funnily enough, the build stage could not read or access it. This time i just let Codepipeline create the S3 bucket and configure its role to interact with it and voila!, it could be some other issue but that would need more testing.

@oOPa
Copy link

oOPa commented May 12, 2021

+1

@oOPa
Copy link

oOPa commented May 23, 2021

I had a similar issue and the solution was to grant permissions via the KMS resource based policy.

@nirojshrestha019
Copy link

I am getting the same error while codepipeline is trying to triger the codebuild. Here is the S3 artifact bucket that I have used:

resource "aws_s3_bucket" "codepipeline_bucket" {
  bucket        = var.codepipeline_bucket_artifact_bucket_name
  acl           = "private"
  force_destroy = true
  server_side_encryption_configuration {
    rule {
      apply_server_side_encryption_by_default {
        sse_algorithm = "AES256"
      }
    }
  }
}

I have used the above encryption for the bucket. Could anyone please help what IAM policy should I add to the IAM user to avoid the error :

AccessDenied: Access Denied
status code: 403, 
request id: R4TTGRC9KXJMWDFE, host id: 
Lk4yDF7ZKrFMyTMdI+ncpsLNtQQBd1Oy5UHs2rpibOuMrPf+pwD76Me9b6kUUxDDGAa5H6RH13w=
 for primary source and source version 
arn:aws:s3:::codepipelinecreatepatients3bucket/codepipeline_create_/source_out/tVhtjC0


This is the Policy that I am currently using for the IAM user of codepipeline


  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect":"Allow",
      "Action": "s3:*",
      "Resource": [
        "${aws_s3_bucket.codepipeline_bucket.arn}",
        "${aws_s3_bucket.codepipeline_bucket.arn}/*"
      ]
    },
  
    {
            "Effect": "Allow",
            "Action": [
                "kms:CreateAlias",
                "kms:CreateKey",
                "kms:DeleteAlias",
                "kms:Describe*",
                "kms:GenerateRandom",
                "kms:Get*",
                "kms:List*",
                "kms:TagResource",
                "kms:UntagResource",
                "iam:ListGroups",
                "iam:ListRoles",
                "iam:ListUsers"
            ],
            "Resource": "*"
        },
    

    {
      "Effect": "Allow",
      "Action": [
        "codebuild:BatchGetBuilds",
        "codebuild:StartBuild"
      ],
      "Resource": "*"
    },

        {
            "Effect": "Allow",
            "Action": [
                "codecommit:*"
            ],
            "Resource": "*"
        },

           {
            "Action": [
                "codebuild:*",
                "codecommit:GetBranch",
                "codecommit:GetCommit",
                "codecommit:GetRepository",
                "codecommit:ListBranches",
                "codecommit:ListRepositories",
                "cloudwatch:GetMetricStatistics",
                "ec2:DescribeVpcs",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeSubnets",
                "ecr:DescribeRepositories",
                "ecr:ListImages",
                "elasticfilesystem:DescribeFileSystems",
                "events:DeleteRule",
                "events:DescribeRule",
                "events:DisableRule",
                "events:EnableRule",
                "events:ListTargetsByRule",
                "events:ListRuleNamesByTarget",
                "events:PutRule",
                "events:PutTargets",
                "events:RemoveTargets",
                "logs:GetLogEvents",
                "s3:GetBucketLocation",
                "s3:ListAllMyBuckets"
            ],
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Action": [
                "logs:DeleteLogGroup"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:logs:*:*:log-group:/aws/codebuild/*:log-stream:*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ssm:PutParameter"
            ],
            "Resource": "arn:aws:ssm:*:*:parameter/CodeBuild/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ssm:StartSession"
            ],
            "Resource": "arn:aws:ecs:*:*:task/*/*"
        }
  ]
}
EOF


@nirojshrestha019
Copy link

nevermind I found the solution. In my case, I had to give s3 access to codebuild in addition to the codepipeline IAM role.

@hari1992-web
Copy link

i am facing an issue cross account codepipeline The scenario is One account to different account in the same region is
working and I am trying this scenario is one account to different account with different region is not working iam getting below error
Insufficient permissions
The service role or action role doesn’t have the permissions required to access the AWS CodeCommit repository named codecommit. Update the IAM role permissions, and then try again. Error: User: arn:aws:sts::117270586458:assumed-role/CrossAccountCodeCommitRole/1622195893168 is not authorized to perform: codecommit:GetBranch on resource: arn:aws:codecommit:ap-southeast-1:117270586458:codecommit
crossregion

@hari1992-web
Copy link

can any body help ???

@ravishtiwari
Copy link

I ran into this error recently, in my case, I was using AWS Code star connection with codepipeline, and it had permission to use the connection, however, codebuild was not having sufficient permission (I was using passing git clone meta information to codebuild). I added required permission to codebuild, and I was able to resolve the issue:

statement {
    actions = [
        "codestar-connections:UseConnection",
        "codestar-connections:GetConnection"
      ]
    resources = [codestarconnections_arn]
  }

However, as a lot of folks have explained, you would run into this issue if your code build does not have permission to the KMS key used to encrypt.

@cobbgcall
Copy link

I read all previous comments, in my case I modified codebuil role adding access to s3 bucket. Now, it is working.
So, I would like to recommend to try using less privileged strategy.
"s3:ListBucket",
"s3:PutObject",
"s3:GetObject"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests