(assets): Allow using exported variables from Assets stage steps

[Hi-Fi]

Describe the feature

Way to refer to variables in latter CodePipeline steps that are exported from Assets stage's steps.

Use Case

This would speed up pipeline run quite a lot, when we can do container security scanning in Asset stage (which is already possible) and get the ID of the scan to be included in latter created documents about the builds.

We can export the value as it's doable in partialBuildSpec, but we can't refer that in latter steps as Asset stage steps doesn't have namespace set.

We can dynamically get the asset steps that build docker images in Synth step with jq and exported variables.

export ASSET_STAGES_EXPORTED_IDS=$(jq -r 'map(.. | select(.Name? and .Name=="Assets") | .Actions[].Name | select(test("DockerAsset.*")) | "#{Assets@\\(.).EXPORTED_SCAN_ID}") | join(",")' cdk.out/${pipelineStackName}.template.json)`,

If there would even be static namespace (in example Assets, it would be possible to use those value later).

Proposed Solution

Export Asset steps' CodeBuildSteps same way as Synth has those exposed so that it's possible to use variable referencing from steps in Asset stage. Kind of issue here could be that it would also require calculation of amount of needed asset steps earlier than now.

Other option would be just hardcode the namespace also to Asset stage steps so exported values from those could manually be referred. If namespace would be as in other Asset@DockerAsset1 etc., still above used way to get the needed steps would be needed. But this would still work.

Other Information

We have CodeBuild projects that CDK created CodePipeline uses deployed to our custom VPC, meaning that provisioning time is minimum around 90 seconds. We also do security scanning of the built image, and image is created at the Asset stage.

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

CDK version used

2.20.0

Environment details (OS name and version, etc.)

Debian bullseye

[Hi-Fi]

Is there some reason why those namespaces just couldn't be added always to steps in Asset stage? As I think more sophisticated way of referring those values might be a bit clumsy due to amount of steps that can be from 1 to n.

[Hi-Fi]

@rix0rrr @corymhall @TheRealAmazonKendra @comcalvi @otaviomacedo

If it's not OK to export variables from Asset steps, what would be the way to run tests that last ~10 seconds, require that DockerImageAssets are created and need to be able to export some value to pipeline for later use?

Pipelines uses already quite heavily those CodeBuilds causing provision time and risk of queuing really show on developer side.

Current approach with CodeBuildStep after Asset stage means, that:

• We provision things around minute (if no queuing)
• We have to determine the image tag from cdk.out to know what image was pushed
• We have to pull image back from ECR where it was pushed
• We have to execute the scanning (this takes around 10 seconds max)

In good case this takes around 3 minutes, but is easily quite a lot longer. So about 18x longer than just executing the thing in asset steps (that produce Docker images). And that's also possible, but then we can't get the value to later part where we need to report the scan that was made, as ID can't be exported from those "magical" Asset stage's steps.

[Hi-Fi]

One workaround would be to have some separate Lambda in the pipeline that would read those exported values (without namespace) directly from steps. Then it can either export those variables for pipeline or use those self.

This can create extra costs, but probably not at least much (probably less than waiting and provisioning same amount of CodeBuilds for quick actions).