cdk.pipelines - Allow the configuration of Caching properties for Synth CodebuildProject

[mrpackethead]

It would be helpful to add the abillity to set caching for the synth code build project in a pipeline. This will help with speed up the build and run times of pipelines. I have achived the desired result by using escape hatches but this is not particaully elegent and could be improved with some minor work.

if 'SynthProjectCachePropertys' in synth_cfg.keys():

            # the pipeline needs to be built bfore you can extract the synth
            # codebuild project. 
            # Everything needs to be done with buildingpipeline prior to this, 
            # as you can't modify it after you build. s
            pipeline.build_pipeline()

            codebuild_cache_bucket = s3.Bucket(self, 'codebuild_cache_bucket',
                auto_delete_objects= True,
                removal_policy= cdk.RemovalPolicy.DESTROY,
                block_public_access= s3.BlockPublicAccess.BLOCK_ALL
            )

            # finally use an escape hatch to manipulate the codebuild
            SynthCodeBuildProject: codebuild.CfnProject = pipeline.synth_project.node.default_child
            SynthCodeBuildProject.ProjectCacheProperty(
                type="S3",
                location = codebuild_cache_bucket.bucket_name,
                modes = synth_cfg['SynthProjectCachePropertys']['modes'],
            )
            # allow the code build project permission.
            codebuild_cache_bucket.grant_read_write(SynthCodeBuildProject)

### Use Case

Speed up the pipeline time. 

[rix0rrr]

What is missing from CodeBuildStep to achieve what you want?

[gshpychka]

@rix0rrr straightforward access to the CodeBuild project. The BuildSpec by itself is not sufficient.

[mrpackethead]

@rix0rrr , a way to set caching for the codebuild project, but actually, as @gshpychka said, just being able to access the codebuild project directly would be very helpul.

[rix0rrr]

Would a single API that specifies both the Cache property in CloudFormation as well as configure the BuildSpec for you be a good idea? Or what that be too limiting for some use case I'm not thinking of right now?

I'm thinking of something like:

new CodeBuildStep('Step', {
  // Local
  caching: CodeBuildCaching.local({
    source: true,
    docker: true,
    paths: ['~/.m2', '~/.npm'],
  }),

  // S3
  caching: CodeBuildCaching.s3(myBucket, {
    prefix: 'buildcache/',
    paths: ['~/.m2', '~/.npm'],
  }),
});

[gshpychka]

@rix0rrr that would cover my use case, for what it's worth.

But access to the codebuild project would still be useful, to get access to other features like logging.

[mrpackethead]

Right now for what i'm doing just being able to set cache setting is helpful. But, i think being able to access the underlying code build project, would be the most flexible solution.. For example I've got something in mind coming up soon, where i'd probably want to attach the project to a vpc, so it can do some lookups of things that are not AWS specific. I want to be able to do a look up of some 'outside world' things that will influence my build. ( in my case, it will be a list of sensors that are part of an IOT installation )

[rix0rrr]

Attaching to a VPC is already possible today, as is influencing the IAM permissions.

By the way, you can always access pipeline.synthProject (cast it to a Project) to change things about it, although those might require overrides.

And for more customized cases, you can implement an ICodePipelineActionFactory that creates the exact CodeBuild project you want and need.

FWIW, the "this code creates mostly what I need except this one property..." problem crops up a lot more in CDK. I don't think the solution should be "let's just keep adding properties everywhere". Although I don't have the full solution yet either...

[gshpychka]

@rix0rrr thanks. Does this mean that CodeBuildCaching is still coming?

[mrpackethead]

FWIW, the "this code creates mostly what I need except this one property..." problem

Yes, murphy's law is that its always that one property that is'tn there. :-) I agree with you, that adding more propertys is not really a scalable solution to the general problem.. If your contruct starts specifiying 'propertys' of a an underlying reoource the construct, ( rather than the resource itself ), then it woudl always be desirable that there is an 'easy' way to acces that resource.. if this coudl be done as a 'general' thing. ( thats a big ask i know ) then all the 'mostly want i need' problems go away.

[bilalq]

FWIW, the "this code creates mostly what I need except this one property..." problem crops up a lot more in CDK. I don't think the solution should be "let's just keep adding properties everywhere". Although I don't have the full solution yet either...

While that's true, the problem is exacerbated by the lack of access to encapsulated resources. Just being able to get a reference to the underlying project would allow users to make tweaks without needing to keep expanding the API of higher level constructs.

Between expanding the API to cover caching use-cases and exposing the underlying project that is encapsulated, which would you prefer @rix0rrr?

I've run into at least half a dozen issues when trying to setup CI/CD pipelines using CodeStar/CodeBuild/CodeArtifacts/CodePipeline. Some problems can be worked around by using CFN escape hatches, but others encapsulate so much that it's hard to get at things and I end up re-implementing constructs entirely.

[johnameyer]

My two cents are that implementing an ICodePipelineActionFactory is a bit heavy handed here if all you want to do is add caching... reimplementing CodeBuildFactory.fromCodeBuildStep is nearly 200 lines of bulk versus adding it alongside the other properties that are just passed along at

aws-cdk/packages/@aws-cdk/pipelines/lib/codepipeline/_codebuild-factory.ts

Lines 269 to 271 in ebba618

	vpc: projectOptions.vpc,
	subnetSelection: projectOptions.subnetSelection,
	securityGroups: projectOptions.securityGroups,

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.