New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
aws-lambda: unable to upload function code asset to cdk asset bucket #23249
Comments
The error message looks a bit misleading, as it can occur for more reasons than the bucket being in the wrong account. This error will occur when you are getting either an Here are a few things I think you can try
|
I'll check your first bulleted suggestion and report back once able. Any tips on filtering the cloud trail event logs for that event would be appreciated as I'm a novice there. I assume I'll be able to use Is the deploy role you mentioned different than the CDK's execution policy? You can see in the bootstrap output that the execution policy is the AdministratorAccess managed policy. And the CDK docs here state that the execution policy is the "deployment role assumed by CloudFormation during deployment of your stacks." I already tried setting those environment variables and removing the |
Yes, there is a |
Ok I think we're getting somewhere. I ran the following to generate in CloudTrail a successful aws s3api get-bucket-location \
--bucket cdk-<random-id>-assets-<account-id>-<region> \
--profile mfa In CloudTrail, I can see the following for that successful api call: "userIdentity": {
"type": "IAMUser",
"principalId": "<principal-id>",
"arn": "arn:aws-us-gov:iam::<account-id>:user/<my-iam-user>",
"accountId": "<account-id>",
"accessKeyId": "<access-key-id>",
"userName": "<my-iam-user>",
"sessionContext": {
"sessionIssuer": {},
"webIdFederationData": {},
"attributes": {
"creationDate": "2022-12-07T21:22:37Z",
"mfaAuthenticated": "true"
}
}
}, Running Lo and behold, that role lacks the {
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:GetObject*",
"s3:GetBucket*",
"s3:GetEncryptionConfiguration",
"s3:List*",
"s3:DeleteObject*",
"s3:PutObject*",
"s3:Abort*"
],
"Resource": [
"arn:aws-us-gov:s3:::cdktoolkit-stagingbucket-cdk-bug",
"arn:aws-us-gov:s3:::cdktoolkit-stagingbucket-cdk-bug/*"
],
"Effect": "Allow"
},
{
"Action": [
"kms:Decrypt",
"kms:DescribeKey",
"kms:Encrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey*"
],
"Resource": "arn:aws-us-gov:kms:<region>:<account-id>:key/AWS_MANAGED_KEY",
"Effect": "Allow"
}
]
} Naturally, I added At any rate, I tried Do you wish to deploy these changes (y/n)? y
CdkBugStack: deploying...
[0%] start: Publishing 6db6c214750fd7227d61d7accfde57987808698c67cb93a0fbd0328d10fc2926:395913319687-<region>
[0%] start: Publishing 81ab06eacb598337b1c4663146a3ef12f7b208990c3f0fa96f2692235a73fdca:395913319687-<region>
[50%] success: Published 81ab06eacb598337b1c4663146a3ef12f7b208990c3f0fa96f2692235a73fdca:395913319687-<region>
[100%] success: Published 6db6c214750fd7227d61d7accfde57987808698c67cb93a0fbd0328d10fc2926:395913319687-<region>
CdkBugStack: creating CloudFormation changeset...
❌ CdkBugStack failed: Error [ValidationError]: S3 error: Access Denied
For more information check http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html
at Request.extractError (/Users/me/.nvm/versions/node/v16.10.0/lib/node_modules/aws-sdk/lib/protocol/query.js:50:29)
at Request.callListeners (/Users/me/.nvm/versions/node/v16.10.0/lib/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/Users/me/.nvm/versions/node/v16.10.0/lib/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/Users/me/.nvm/versions/node/v16.10.0/lib/node_modules/aws-sdk/lib/request.js:686:14)
at Request.transition (/Users/me/.nvm/versions/node/v16.10.0/lib/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/Users/me/.nvm/versions/node/v16.10.0/lib/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /Users/me/.nvm/versions/node/v16.10.0/lib/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request.<anonymous> (/Users/me/.nvm/versions/node/v16.10.0/lib/node_modules/aws-sdk/lib/request.js:38:9)
at Request.<anonymous> (/Users/me/.nvm/versions/node/v16.10.0/lib/node_modules/aws-sdk/lib/request.js:688:12)
at Request.callListeners (/Users/me/.nvm/versions/node/v16.10.0/lib/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
code: 'ValidationError',
time: 2022-12-07T21:46:54.362Z,
requestId: '7d48fd5d-288b-40cc-8a3d-504d2af8a7e0',
statusCode: 400,
retryable: false,
retryDelay: 855.9092614166715
}
❌ Deployment failed: Error: Stack Deployments Failed: ValidationError: S3 error: Access Denied
For more information check http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html
at deployStacks (/Users/me/.nvm/versions/node/v16.10.0/lib/node_modules/aws-cdk/lib/deploy.ts:61:11)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at CdkToolkit.deploy (/Users/me/.nvm/versions/node/v16.10.0/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:338:7)
at initCommandLine (/Users/me/.nvm/versions/node/v16.10.0/lib/node_modules/aws-cdk/lib/cli.ts:364:12)
Stack Deployments Failed: ValidationError: S3 error: Access Denied
For more information check http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html I literally have to walk out the door right now, but I imagine the next debugging step is to lookup that requestId of |
Looking at that request will probably give you some more insights yeah, but I'm confused at why the file publishing role is being used here. I'm also not quite sure why the resource policies have I can help investigate more once you get back to me with the findings of that new request 🙂 |
I found the failed deployment in CloudTrail as a {
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"cloudformation:CreateChangeSet",
"cloudformation:DeleteChangeSet",
"cloudformation:DescribeChangeSet",
"cloudformation:DescribeStacks",
"cloudformation:ExecuteChangeSet",
"cloudformation:CreateStack",
"cloudformation:UpdateStack"
],
"Resource": "*",
"Effect": "Allow",
"Sid": "CloudFormationPermissions"
},
{
"Condition": {
"StringNotEquals": {
"s3:ResourceAccount": "<account-id>"
}
},
"Action": [
"s3:GetObject*",
"s3:GetBucket*",
"s3:List*",
"s3:Abort*",
"s3:DeleteObject*",
"s3:PutObject*"
],
"Resource": "*",
"Effect": "Allow",
"Sid": "PipelineCrossAccountArtifactsBucket"
},
{
"Condition": {
"StringEquals": {
"kms:ViaService": "s3.<region>.amazonaws.com"
}
},
"Action": [
"kms:Decrypt",
"kms:DescribeKey",
"kms:Encrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey*"
],
"Resource": "*",
"Effect": "Allow",
"Sid": "PipelineCrossAccountArtifactsKey"
},
{
"Action": "iam:PassRole",
"Resource": "arn:aws-us-gov:iam::<account-id>:role/cdk-<random-identifier>-cfn-exec-role-<account-idd>-<region>",
"Effect": "Allow"
},
{
"Action": [
"cloudformation:DescribeStackEvents",
"cloudformation:GetTemplate",
"cloudformation:DeleteStack",
"cloudformation:UpdateTerminationProtection",
"sts:GetCallerIdentity",
"cloudformation:GetTemplateSummary"
],
"Resource": "*",
"Effect": "Allow",
"Sid": "CliPermissions"
},
{
"Action": [
"s3:GetObject*",
"s3:GetBucket*",
"s3:List*"
],
"Resource": [
"arn:aws-us-gov:s3:::cdktoolkit-stagingbucket-cdk-bug",
"arn:aws-us-gov:s3:::cdktoolkit-stagingbucket-cdk-bug/*"
],
"Effect": "Allow",
"Sid": "CliStagingBucket"
},
{
"Action": [
"ssm:GetParameter"
],
"Resource": [
"arn:aws-us-gov:ssm:us-gov-west-1:<account-id>:parameter/cdk-bootstrap/<random-id>/version"
],
"Effect": "Allow",
"Sid": "ReadVersion"
}
]
} A few things strike me as odd with that policy:
Before I change the buckets in that policy, I want to see what actually happened when I bootstrapped. So I re-ran StagingBucket:
Type: AWS::S3::Bucket
Properties:
BucketName:
Fn::If:
- HasCustomFileAssetsBucketName
- Fn::Sub: ${FileAssetsBucketName}
- Fn::Sub: cdk-${Qualifier}-assets-${AWS::AccountId}-${AWS::Region} To make sure I didn't bootstrap with an older version of the CDK, I can verify that ➜ cdk bootstrap --template template.yaml --profile mfa
Using bootstrapping template from template.yaml
⏳ Bootstrapping environment aws://<account-id>/<region>..
Trusted accounts for deployment: <account-id>/
Trusted accounts for lookup: <account-id>/
Execution policies: arn:aws-us-gov:iam::aws:policy/AdministratorAccess
✅ Environment aws://<account-id>//<region> bootstrapped (no changes). That last note of Finally, replacing: "Resource": [
"arn:aws-us-gov:s3:::cdktoolkit-stagingbucket-cdk-bug",
"arn:aws-us-gov:s3:::cdktoolkit-stagingbucket-cdk-bug/*"
], with: "Resource": [
"arn:aws-us-gov:s3:::cdk-<random-identifer>-assets-<account-id>-<region>",
"arn:aws-us-gov:s3:::cdk-<random-identifer>-assets-<account-id>-<region>/*"
], in that IAM policy results in a successful ✅ CdkBugStack
✨ Deployment time: 46.94s 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 |
I'm really glad to hear you got this working! Based on the existing policy you shared, it really looks like this stack was originally bootstrapped with Legacy CDK v1 bootstrapping. Why this would result in no changes on your next attempt, I'm not sure, but I'm glad you got this working! |
|
Describe the bug
I've got a few CDK apps written in v1 as well as a few written in v2. Today, when creating a new v2 CDK project from scratch, I am unable to deploy due to an error with uploading the CDK asset for the Lambda function code. The relevant error snippet is:
Expected Behavior
I expected
cdk deploy
to utilize the CDK resources (staging and assets s3 buckets) after a successfulcdk bootstrap
.Current Behavior
The full error of my
cdk deploy
command, with my personal info, account number and region redacted:Reproduction Steps
I created a whole new CDK v2 project from scratch to reproduce this. Here are those steps:
First, install deps and create the new project
In
./lib/cdk-bug-stack.ts
I added a lambda function usinglambda.Code.fromAsset()
It references some Python code in
./app/main.py
:And in
./bin/cdk-bug.ts
I added theenv
prop to set my AWS Account and Region:My AWS credentials are configured with a named profile, defined in
~/.aws/credentials
:As hinted in the profile name, my organization requires MFA, so those values in my credentials file were created using the following:
I know this
mfa
profile works because of the following:And I can even use this named profile to list the contents of my CDK assets bucket (created by a previous cdk app) using the S3 cli:
Now here's where it gets weird.
If I bootstrap this project with the following:
I get the following success message:
As you can see, the execution policy is
AdministratorAccess
. That is the same managed policy attached to my IAM user.But then on deployment, using
cdk deploy --profile mfa
, I get the following error:I found others with a similar error message in 6808, but the root cause for them seemed to be a lack of permissions in the IAM execution policy used by the CDK. That is, they needed
s3:GetObject
,s3:PutObject
,s3:ListBucket
, ands3:GetBucketLocation
on resourcecdk-*-assets-*-*
. This shouldn't be an issue for me since thatAdministratorAccess
policy covers everything:Just to test if my
mfa
profile does indeed have permissions, I tried uploading the python script myself to that assets bucket:This works. And I can - and have - performed all sorts of other aws cli calls using
--profile mfa
in the past.Possible Solution
Not sure.
Additional Information/Context
Operating in AWS gov-cloud.
CDK CLI Version
2.53.0 (build 7690f43)
Framework Version
No response
Node.js Version
16.10.0
OS
macOS Catalina 10.15.6
Language
Typescript
Language Version
~3.9.7
Other information
No response
The text was updated successfully, but these errors were encountered: