Bootstrap bucket/ECR repository garbage collector

[markcarroll]

Our CDK Bootstrap staging bucket on S3 now has gigabytes of assets in there, most of which are either old versions of current stacks, or for stacks that have long since been deleted.

How do we go about cleaning these up?

Does the CDK have any capability to clean up out of date or no longer needed assets? If not, it seems like it should. This is starting to incur noticeable costs for us.

[arpowers]

just run a rm -rf at some point in your lifecycle.

[markcarroll]

on the S3 staging bucket? My understanding is that the asset files are referred to by CloudFormation.

[arpowers]

Ah I thought you were talking about cdk.out... guessing this is in cdk/pipeline?

[hoegertn]

I think markcarroll is talking about the cdk bootstrap bucket that holds the assets.

[markcarroll]

correct. thanks for the terminology correction, I have updated my question to clarify.

[SomayaB]

Related to #986

[rix0rrr]

Looks like we don't yet have a feature request for the Assets garbage collector. Repurposing this.

The unfortunate answer is that we currently don't have a good solution for you.

[luxaritas]

Per the above issue cross reference, this is a big issue for containers which can be individually multiple GBs

[elhedran]

It seems to me this would be easier if you could have a 1-1 mapping of bootstrap stacks to deploy stacks - could then just delete any asset not deployed in this deployment upon success (cleanup), anything in the repo/s3 bucket that isn't part of the current list of assets from the stacksynth can go. Overly simplistic I know - but how much value is there by sharing the bootstrap stack across deployment stacks, because I'm not sure I see it?

As near as I can tell you could do it for ecr and s3 via tagging as well, (tag current on deploy, tag prior on cleanup) and have a lifecycle configured for the ecr/s3 based on tag - only the current IStackSynthesizer doesn't seem to be given access to the locations of the prior assets no longer used so couldn't change their tags. If it had two functions retireFileAsset(asset: FileAssetLocation) and retireDockerImageAsset(asset: DockerImageAssetLocation) it should be possible to implement garbage collection external to the cdk (e.g. custom stack-synth) - that is assuming that the cdk has some record of the prior stacks assets to do set operations on against the current stack assets.

[milesgranger]

It seems, propagating the tags for a given deployment to the ECR images and s3 assets would be enough for users to generate their own garbage collection.

We tag our deployments/assets using standard semantic versioning, and can 'easily' remove objects past a given version. However, it doesn't appear assets loaded into the bootstrap s3 bucket get the same tags as resources in the stack; which I find slightly weird. Please let me know if I've missed something here. :)

[gbvanrenswoude]

Decent attempt at keeping assets under control while AWS is working on this: https://github.com/jogold/cloudstructs/blob/master/src/toolkit-cleaner/README.md

[peterwoodworth]

Thanks all for asking support for this feature, we're aware it's desired and have put in some progress towards implementing this. I'll be closing this however in favor of #6692 so we can keep discussion in one place

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.