--flatten behavior

[till]

Before I create (more) issues, I wanted to ask if someone could share how --flatten on buildpacks and builder is supposed to work. I've read the RFC and read PRs, etc.. But I don't have a great idea yet.

Some context:

• my builder is based on paketo's latest (jammy) builder
• I am adding paketo-buildpacks/dotnet on top of that
• I am also including some downstream buildpacks (that we wrote inhouse)
• my overall goal is to optimize for the 127 layer limit (I am at 119)

So far, I've observed and tried the following:

flatten a composite buildpack

Flattening a composite buildpack makes my builder grow in size rapidly. It seems that the deduplication of buildpacks no longer works. Or maybe it is not supposed to anymore?

For additional context, with monthly updates to the builder before we very hovering around 900 MB, then had a 1.1 GB increase (due to our own buildpacks and their own dependencies) and now since I've attempted to flatten various buildpacks I am already at 1.27 GB.

I think this should be added to the RFC as a drawback. But not entirely sure what the process is.

fatten a composite buildpack, which "shares" dependencies with other composite buildpacks

This is when I tried to flatten paketo-buildpacks/nodejs.

Because this composite buildpack "shares" dependencies (e.g. paketo-buildpacks/node-engine) with other composite buildpacks, the deduplication didn't work.

What I mean by "sharing" is, that its dependencies are also dependencies in other buildpacks/plans.

The result is that the layers are growing exploding (despite trying the opposite with flatten).

flatten a builder

Before I tried to flatten composite buildpacks, I tried flattening the builder. This recently stopped working and I can't figure out why.

I constructed a very long --flatten to include all buildpacks and potential versions. This now leads to the builder/Docker image containing duplicate paths. Which then fails when I docker pull from a registry.

If I run pack without --publish, then it fails the pack builder create (when it's trying to add the image to the local dockerd).

It also works sometimes, which means pack builder create is not deterministic?

More outlined here: buildpacks/pack#2429