-
Notifications
You must be signed in to change notification settings - Fork 3.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
(aws-codepipeline): Asset from BucketDeployment not deployed to cross-account bucket #20609
Comments
Just found my old report that mentioned exactly the scenario - #13940 - which should be now automatically fixed by CDKv2 migration but it seems it's not. |
Which part of this exactly is cross-account? |
Sorry this was indeed not visible from the code above, so here is how the pipeilne is constructed export class DynamicPipelineConstruct extends Construct {
this.pipeline = new Pipeline(this, 'pipeline', {
pipelineName: `${id}-pipeline`,
artifactBucket: this.artifactBucket,
role: this.devOpsPipelineRole,
});
....
}
export class CdkBuildConstruct extends Construct {
artifact: Artifact;
action: CodeBuildAction;
constructor(scope: Construct, id: string, props: CdkConstructProps) {
super(scope, id);
let buildCommand = 'npm run -- cdk synth "*" -c dev_mode="false"';
this.artifact = new Artifact(`${props.assetPrefix}-${id}`);
const cdkBuild = new PipelineProject(this, id, {
projectName: `${props.assetPrefix}-${id}`,
environment: { buildImage: LinuxBuildImage.STANDARD_5_0 },
encryptionKey: props.encryptionKey,
role: props.role,
buildSpec: BuildSpec.fromObject({
version: '0.2',
phases: {
install: {
'runtime-versions': {
nodejs: 14,
},
commands: [
'npm install -g aws-cdk',
'cdk --version',
'aws codeartifact login ...,
'cd cdk',
'npm ci',
],
},
build: {
commands: ['npm run build', buildCommand],
},
},
artifacts: {
'base-directory': `cdk/cdk.out`,
files: ['**/*'],
},
}),
});
this.action = new CodeBuildAction({
runOrder: props.runOrder,
actionName: id,
project: cdkBuild,
input: props.sourceInput,
outputs: [this.artifact],
role: props.role,
});
}
export class ServicesPipelineConstruct extends DynamicPipelineConstruct {
const templatePath = `${props.deployedStackName}.template.json`;
const cdkbuild = new CdkBuildConstruct(this, 'cdk', {
runOrder: 1,
encryptionKey: this.encryptionKey,
assetPrefix: this.assetPrefix,
sourceInput: this.sourceOutput,
role: this.pipeline.role,
});
}
const lambdaArtifact = new Artifact(`${this.assetPrefix}-lambda-artifact`);
const lambdaBuild = new PipelineProject(this, 'LambdaBuild', {
projectName: `${this.assetPrefix}-lambda-build`,
environment: { buildImage: LinuxBuildImage.STANDARD_5_0 },
buildSpec: BuildSpec.fromSourceFilename('lambda/buildspec.yaml'),
encryptionKey: this.encryptionKey,
role: this.pipeline.role,
});
const lambdaCodebuildAction = new CodeBuildAction({
runOrder: 1,
actionName: `Lambda`,
project: lambdaBuild,
input: this.sourceOutput,
outputs: [lambdaArtifact],
role: this.pipeline.role,
});
this.pipeline.addStage({
stageName: 'Build',
actions: [
lambdaCodebuildAction,
cdkbuild.action,
],
});
// here the cross-account deployment step that fails because cdk assets are not present in the shared bucket
this.pipeline.addStage({
stageName: 'Deployment-INTEGRATION',
actions: [
new CloudFormationCreateUpdateStackAction({
runOrder: 1,
actionName: 'DeployCF',
stackName: props.deployedStackName,
adminPermissions: false,
role: this.integrationDevOpsRole,
deploymentRole: this.integrationChangeSetRole,
parameterOverrides: {
s3LambdaCodeBucketName: lambdaArtifact.bucketName,
s3LambdaCodeBucketKey: lambdaArtifact.objectKey,
},
extraInputs: [lambdaArtifact],
templatePath: cdkNoMonitoring.artifact.atPath(`${templatePath}`),
cfnCapabilities: [CfnCapabilities.NAMED_IAM, CfnCapabilities.AUTO_EXPAND],
})
]});
} The mentioned BucketDeployment is added in the CDK stack of the app itself, so not in the pipeline stack above, but inside of the source code that is built with the |
Coming to think about it after weekend, maybe I am missing a deployment step for the shared assets part?
I don't see it anywhere in the docs, is there a special step needed to actually deployed the shared asset to the central devops account bootstrap shared bucket that will later be re-used by the pipeline during deployment in other accounts? (loud thinking) const bootstrapBucket = Bucket.fromBucketAttributes(this, 'cdkBucket', {
bucketArn: Fn.importValue(DefaultStackSynthesizer.DEFAULT_FILE_ASSETS_BUCKET_NAME),
encryptionKey: this.encryptionKey,
});
const deployCdkAssetsAction = new S3DeployAction({
actionName: 'S3Deploy',
input: cdkNoMonitoring.artifact,
bucket: bootstrapBucket,
runOrder: 2,
}); On the other hand, since the pipeline already has an Summing my thoughts up to one sentence: It seems that |
OK, I'm that close to making it work as I described above, but before I describe my findings and potential bugs, let me ask a question to clarify if I'm not missing something. After bootstrapping the "central" environment, I see that the cdk bootstrap bucket does not have any policies attached to it.
I see that on the {
"Action": [
"s3:GetObject*",
"s3:GetBucket*",
"s3:List*"
],
"Resource": [
"arn:aws:s3:::cdk-hnb659fds-assets-TARGET-eu-west-1",
"arn:aws:s3:::cdk-hnb659fds-assets-TARGET-eu-west-1/*"
],
"Effect": "Allow",
"Sid": "CliStagingBucket"
}, but the TARGET here is actually wrong as I would expect DEVOPS there as being the central repository trusted account where the pipeline actually runs and builds the artefacts. So am I confusing something here? If that is the case, could you point me to some documentation where this is described and to the relevant code that is actually doing the push of the artefact during pipeline build from the central account to the target account? On top of that, the same is with the KMS key: the bucket in the DEVOPS account is provisioned to use the alias/s3 key that is said not to support cross-account deployment, so why in a cross-account scenario a CMK is not created by default and appropriate cross-account policies are not added to it? I have a feeling the new bootstrap is still not supporting cross-account deployment out-of-the-box (but I'm sure I can force it to do so, but first I'd like to check if I'm not reinventing the wheel :D) |
@peterwoodworth, looking at the documentation, I'm getting even more confused:
But the CMK is not created and if I check the CDKToolkit CloudFormation stack that was upgraded with
and if I check the Obviously, that What's wrong here? is the bootstrap buggy to not provision the CMK+S3 bucket correctly with the CMK? |
Hey, thanks for all this info @rantoniuk. I'll take a look at this on monday 🙂 |
Sorry I haven't gotten to this yet, will do my best by tomorrow |
@peterwoodworth I would be grateful for any reply at least on my assumptions as I don't know which way to go. One way is that I hack my way through this by manually fixing bootstrap issues and by changing the used bucket and encryption key and summarise it in a bullet-pointed list of bugs for bootstrap. However if you confirm my suspicions or bad assumptions even in a one sentence summary then I can either redesign the solution to comply with bootstrap's way of working if it's actually correct :) |
I decided to give it a go again, I'll describe what I achieved. To sum up the scenario:
The problem with the old approach is that The new setup is:
The CDK code was rewritten to this: new BucketDeployment(this, 'bd', {
sources: [Source.asset(path.join(__dirname, 'static'))],
destinationBucket: someBucket,
destinationKeyPrefix: 'static',
});
const nodeLayer = new LayerVersion(this, 'lv', {
code: Code.fromAsset(path.join(__dirname, '../../lambda/layer/')),
});
const lambdaCode = Code.fromAsset(path.join(__dirname, '../../lambda/dist/src/')); Now we get to the problem description: The above works perfectly fine locally: after doing The updated pipeline definition looks like this:
new PipelineProject(this, id, {
encryptionKey: props.encryptionKey,
role: props.role,
buildSpec: BuildSpec.fromObject({
version: '0.2',
phases: {
install: {
'runtime-versions': { nodejs: 14 },
commands: [
'aws --version',
'aws codeartifact login --tool ...',
'npx cdk --version',
'cd cdk && npm ci && cd $CODEBUILD_SRC_DIR',
'cd lambda && npm ci && cd $CODEBUILD_SRC_DIR',
'cd lambda/layer/nodejs && npm ci && cd $CODEBUILD_SRC_DIR',
],
},
build: {
commands: [
// lambda build needs to be first as cdk builds needs the produced assets
'cd lambda && npm run build && cd $CODEBUILD_SRC_DIR',
// nodejs doesn't need build command, it's node_modules dependencies only
"cd cdk && npx cdk synth '*' && cd $CODEBUILD_SRC_DIR",
],
},
},
artifacts: {
// files: ['cdk/cdk.out/**/*', 'lambda/dist/src/**/*', 'lambda/layer/**/*'],
'base-directory': 'cdk/cdk.out',
files: ['**/*'],
},
}),
}); The result of the above when the pipeline runs is that it produces ONE artifact with all the assets from cdk.out zipped and puts it correctly in the cdk-hnb659fds-assets-PIPELINE-eu-west-1 bucket. The next step in the pipeline is to deploy: new CloudFormationCreateUpdateStackAction({
runOrder: 1,
actionName: 'DeployCF',
stackName: props.deployedStackName,
adminPermissions: false,
role: this.integrationDeployRole,
deploymentRole: this.integrationCfnExecRole,
// getting rid of those as they shouldn't be needed anymore
// parameterOverrides: {
// s3LambdaLayerCodeBucketName: lambdaLayerArtifact.bucketName,
// s3LambdaLayerCodeBucketKey: lambdaLayerArtifact.objectKey,
// s3LambdaCodeBucketName: lambdaArtifact.bucketName,
// s3LambdaCodeBucketKey: lambdaArtifact.objectKey,
// },
// extraInputs: [lambdaArtifact, lambdaLayerArtifact],
templatePath: buildNoMonitoring.artifact.atPath(`${templatePath}`),
cfnCapabilities: [CfnCapabilities.NAMED_IAM, CfnCapabilities.AUTO_EXPAND],
}), The deployment fails on S3 No such object, so let's see what's happening:
This obviously will not work, because:
If I would be able to somehow Hopefully that brings a refresher to the problem and gives you the full insight. I have resolved all S3 access issues and KMS problems (with the custom CMK) and stumbled on some bugs, but I will report them separately in order not to overload this thread. Note: I'm not using cdkv2 Modern API, nor new Stage/addApplicationStage() etc. Thanks a lot for help on this. |
Hi @peterwoodworth, I created a [reproduction code](https://github.com/rantoniuk/aws-cdk-repro] for this so you can easily see the concept and reproduce it easily. Of course, you'll need to provision the roles and buckets, but other than this it should work out of the box. Here is what is happening here:
What is working:
What is not working:
However, if you look into cdk.out/TestStack.assets.json, you'll see they are properly referenced to get fetched from the bootstrap's bucket. I suspect though that |
Found your issue, and I am having a similar problem Will review the thread above. |
@peterwoodworth were you able to reproduce using the code attached? |
I just found #9917 and I think this is the whole reason this is not working out of the box... |
Hi, CDK Pipelines (https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.pipelines/README.html) supports bucketdeployment and also other dynamically linked assets, like lambda functions, nested stacks etc; it is very simple to use. It links dynamically the assets in an automatically generated step in the beginning of the pipeline. Bucket deployment contributes several assets (probably one for the bucket content, one for the upload function and one for the cloudfront invalidation one). I am attaching a screenshot of the assets of a rather complex stack to clarify this. It contains 4 nested stacks, each of which has one bucket deployment, some dynamically bundled lambda functions with the NodeJsFunction etc... I hope this helps |
Hi @rantoniuk I am looking into this issue now. Please let me confirm this again: As the code you provided is a little bit out-of-dated. Can you share your code snippet that still relevant today so I can copy and paste into my IDE and And: Is this still the error message?
|
This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled. |
Comments on closed issues and PRs are hard for our team to see. |
Describe the bug
This is going to be tricky to describe without all the code, but I'll do my best.
I have a pipeline that is using Pipeline from CodePipeline, CDKv2, current version.
Since all components were upgraded to CDKv2, I'm assuming it's using newStyleSynthesis by default and I can see some hints of that in the generated CloudFormation template (checking for Bootstrap version, referencing the bootstrap S3 asset bucket).
Now, there is a deployment step defined like this:
In the cdkNoMonitoring Stack there is a definition of:
and that step fails in the cross-account target AWS account deployment, because:
And indeed, when I check the cdkNoMonitoring CloudFormation template, I can see that CDK generates a reference to the S3 asset bucket on the pipeline account correctly:
If I check the artefact of the cdkNoMonitoring build, I can see that the
asset.02927fd0ce5bb130cbc8d11f17469e74496526efe5186a9ab36e8a8138e9a557
is there.However, that asset is not being indeed deployed to the
cdk-XXX-assets-${AWS::AccountId}-${AWS::Region}
and if I understand correct, that's the moment when that asset should be placed in that cross-account S3 bucket.I know I could do probably the magic with parameterOverrides but I thought that CDKv2 and new bootstrap solved this usecase out of the box, doesn't it?
Am I missing some step? property?
Expected Behavior
as above
Current Behavior
as above
Reproduction Steps
as above
Possible Solution
No response
Additional Information/Context
No response
CDK CLI Version
2.26.0 (build a409d63)
Framework Version
2.27.0
Node.js Version
14.19.3
OS
MacOS
Language
Typescript
Language Version
No response
Other information
No response
The text was updated successfully, but these errors were encountered: