CNB interoperability challenges and opportunities after removal of Stacks from the spec

[dzuelke]

Context

As the CNB spec removes the concept of stacks, buildpacks will eventually (once the deprecation turns into a removal in a future spec revision) no longer be able to definitively express that they require e.g. Heroku-22, a "stack" that is a specific build of Ubuntu 22.04 with a known set of libraries available for buildpacks' binaries to dynamically link against:

[[stacks]]
id = "heroku-22"

Instead, only the distribution name and version (besides base OS flavor and architecture) are exposed by the lifecycle and are to be specified as targets; in Heroku's case, for Heroku-22, this would be "ubuntu" and "22.04"):

[[targets]]
os = "linux"
arch = "amd64"
[[targets.distributions]]
name = "ubuntu"
versions = ["22.04"]

This gives a certain basic guarantee for package and library versions, if installed (assuming they're from the official distribution sources), but not whether they're available on the given build and run images in the first place.

Many buildpacks bundle or load pre-built packages for e.g. language runtimes, which contain executables and shared objects that link against shared objects from the environment they were built on:

$ ldd .heroku/php/lib/php/extensions/no-debug-non-zts-20220829/imap.so 
	libc-client.so.2007e => /lib/libc-client.so.2007e (0x0000004001865000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x0000004001974000)
	libpam.so.0 => /lib/x86_64-linux-gnu/libpam.so.0 (0x0000004001b9c000)
	libssl.so.3 => /lib/x86_64-linux-gnu/libssl.so.3 (0x0000004001bae000)
	libcrypto.so.3 => /lib/x86_64-linux-gnu/libcrypto.so.3 (0x0000004001c52000)
	libgssapi_krb5.so.2 => /lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x0000004002094000)
	libkrb5.so.3 => /lib/x86_64-linux-gnu/libkrb5.so.3 (0x00000040020ea000)
	/lib64/ld-linux-x86-64.so.2 (0x0000004000000000)
	…

Dynamic linking is desirable, both for efficiency (archive size) and security (rebasing updates linked libraries) purposes.

A standard e.g. Ubuntu 22.04 installation is not guaranteed to have these libraries; libc-client.so.2007e is from a package that's explicitly installed during the build of the Heroku-22 stack image.

In the future, CNBs will therefore have to handle the possibility of missing libraries for their pre-built language runtimes and other packages, as they may be executed in a builder that uses base images different from those provided by the buildpack vendor.

While there may be a $CNB_TARGET_ID environment variable present during the detect and build phases, it is not (and should not be) possible to define this ID as part of the [[targets]] list in a buildpack.toml for the purpose of "matching" buildpacks against "stacks".

The resulting "looser" contract makes it easier to combine buildpacks that currently explicitly depend on different stacks - instead, they will declare compatibility with certain OS distributions and versions.

The variable is optional, however, and, again, different vendors' builders and/or base images will have different values. For buildpacks to be interoperable, they must rely solely on distribution name and version e.g. when determining which pre-built language runtime binaries to download.

But then, as stated before, there are no guarantees for a buildpack that the dynamic libraries these binaries link against are actually present on the "ubuntu"/"22.04" image it's is currently running on.

There are a few possible approaches for dealing with this inside buildpacks. I am listing them below for the purpose of discussion, in the hope that in the not-too-distant future, these can be incorporated into the buildpack authors' guide documentation.

All of the approaches below assume that a buildpack author is aiming for interoperability with other buildpacks, builders, and base images. In cases where somebody wants to explicitly throw together a custom buildpack to just work with a particular known stack in a controlled environment, none of this is necessary, of course. Furthermore, "utility buildpacks" that do not vendor any kind of self-compiled binaries that use dynamic linking can generally ignore all of this, as well.

Options for Buildpack authors

In ascending order of effort-to-implement ;)

0. Just ignore it all

"Hope is not a strategy", they say, but for buildpacks that do not vendor anything, or that are supposed to be used in a tightly controlled environment, not targeting anything in particular and "just knowing" that stuff will work, is perfectly fine.

1. Re-purpose $CNB_TARGET_ID

This is effectively the "do nothing" approach that fails a bit more gracefully. Use the value of $CNB_TARGET_ID during build to decide "yup, my built stuff works with that", and warn or fail, or even bail out early during detection*.

In most cases, this is not a desirable option, as it results in effectively no interoperability - a Heroku buildpack can't run in a Paketo builder or v.v., and the result is exactly the same tight coupling that we have at present with $CNB_STACK_ID.

2. Rely on target OS, architecture, distribution name, and distribution version

Instead of using a stack or target ID as the main identifier, use the CNB_TARGET_* variables, for determining compatibility, and/or e.g. what pre-built binaries to vendor, for instance:

$ curl "https://my-happy-package-repo.s3.amazonaws.com/dist/${CNB_TARGET_OS}-${CNB_TARGET_ARCH}/${CNB_TARGET_DISTRO_NAME}-${CNB_TARGET_DISTRO_VERSION}/ruby-3.4.5.tar.gz

And then...

2.A: "Hail Mary"

Then, hope it "just works", because someone took care to ensure the packages for the buildpacks in use are on the base images.

Again, fine for cases where a user controls the entire "stack" of builder, base images, and buildpacks. Effectively similar to Option 0.

2.B: sanity check the binaries

For any downloaded executables and shared objects, ldd them to see if any dynamic NEEDED entries are missing, and alert the user, advising them to use e.g. https://packages.ubuntu.com/search?searchon=contents&keywords=libc-client.so.2007e&mode=exactfilename&suite=jammy&arch=amd64 to figure out what package is missing from their base image.

Better user experience, although users' hands might be tied in cases where they cannot easily change the builder or its base images.

If $CNB_TARGET_ID matches a specific known/expected value, a buildpack may skip this step. See notes on $CNB_TARGET_ID namespacing below.

Buildpacks could also only warn, rather than fail, on missing shared libraries that might not be essential (think a PHP runtime builds' shared ext-imap module not finding its libc-client.so.2007e - maybe the code doesn't even need and load the ext-imap extension into core).

3. Rely on all from 2. plus available libraries

Effectively, run ldconfig -p from the buildpack to get the full list of dynamic libraries that ld.so knows about, and use that list to figure out ahead of time whether pre-built binaries will even work on the current base image.

This requires maintaing appropriate metadata for built binaries (as in, full list of shared libraries they depend on), so it's considerably more effort than either of option 2.

If $CNB_TARGET_ID matches a specific known/expected value, a buildpack may skip this step. See notes on $CNB_TARGET_ID namespacing below.

Afterwards...

3.A: fail early

If the packages on the image aren't sufficient, fail with an explanation of what's missing, similar to option 2.B.

3.B: choose a "simpler" build

Of course, buildpacks could prepare several builds of language runtimes or other software, with different "levels" of dynamic libraries they link against. A buildpack could install a more "minimal" build of a program, which might be enough for the user.

Thoughts on $CNB_TARGET_ID "namespacing"

For effective interoperability, the value of $CNB_TARGET_ID should be required, by the spec, to be namespaced, as buildpacks will likely use them to short-circuit any "stack" sanity checks.

Otherwise, whoever "squats" an identifier like "minimal" or similar first makes it unusable for any other buildpacks - imagine both Paketo and Heroku having "full-sized" images based on Ubuntu 22.04, but with different libraries... if they both called themselves "full", they'd be indistinguishable, and a buildpack could not infer anything about their guaranteed contents this way.

Reverse-domain notation comes to mind as a possible simple pattern, think com.heroku.base-images.run or io.paketo.images.run-minimal.

This should also be an unversioned "flavor" or "variant" identifier, because the version (think 20.04/"bionic", 22.04/"jammy") is already encoded in $CNB_DISTRIBUTION_(ID|VERSION). Maybe $CNB_TARGET_VARIANT or another name is an even better choice for this variable than $CNB_TARGET_ID?

Long-term, e.g. the Heroku buildpacks might then whitelist e.g. "com.heroku.base-images.run" and "io.paketo.images.run-full" inside the buildpack logic, and declare ubuntu and 20.04, 22.04, 24.04 as supported in targets.distributions.

The spec might also benefit from some guidance on what a buildpack should do if the target ID is of an unknown value (probably the same as if it is absent... best effort, as per the possible approaches outlined above).

A note on ABI compatibility

In the list of options above, it is not possible to rely solely on the target OS and architecture plus the available libraries, and to ignore the distribution name and version, if downloaded programs dynamically link against shared libraries on the system.

The reason for that is that ABI versions (e.g. the 6 in libstdc++.so.6) only guarantee forward compatibility at runtime. A program compiled against an old version of a library will still execute fine when dynamically linking against a much newer version (but same ABI version number) of the library.

However, if the program is compiled against a newer version of the library, and then executed against an older version, it is likely to have, via #ifdef macros etc, used newly added symbols from the newer version of the library during compilation, which are not present on an older version.

For example, a program that links against libpq.so.5, built on Debian "bookworm", links against symbols from version 15.3 of libpq. When executed on a Ubuntu "jammy" system, where libpq.so.5 is also installed, this will then not work - the libpq version on that Ubuntu version is 14.8.

As the combinations of exact library versions across Linux distributions and versions are effectively infinite, this means that, in practice, distributions and their versions have to be targeted precisely, both during compilation and when later "vendoring" them inside buildpacks.

[natalieparellano]

Cross posting for visibility: https://github.com/orgs/paketo-buildpacks/discussions/191

[dmikusa]

This gives a certain basic guarantee for package and library versions, if installed (assuming they're from the official distribution sources), but not whether they're available on the given build and run images in the first place.

I would disagree unless the name and versions combine to point to a specific tag of an image, but then you don't have the repo to know where the image resides.

The reason I disagree is that there is no standard install for Ubuntu or any other Linux version. There are so many different ways to do an install of Linux, pull in packages, strip out packages, I don't think that provides much useful information. At least, not when it comes to knowing what's installed in the image.

While there may be a $CNB_TARGET_ID environment variable present during the detect and build phases, it is (and should not be) possible to define this ID as part of the [[targets]] list in a buildpack.toml for the purpose of "matching" buildpacks against "stacks". Such a "looser" contract makes it easier to combine buildpacks that currently explicitly depend on different stacks.

I don't like this option, but I think it's probably the only want to actually have a strong guarantee. If you are making a builder and a set of buildpacks, you can coordinate between the two. If you are making a buildpack targeting a well known builder/base images, then you can target it as well.

As mentioned, it does tightly couple the builder and buildpacks, which isn't great. I could definitely see warning instead of a hard fail if the targets don't match. It also makes me wonder why even change when we have basically the same thing right now with stacks.

2.B: sanity check the binaries

This doesn't work because as a buildpack author you can only sanity check what's in the build image. You could make sure all the required build tools are present, but you can't see the run image so you can't check what's going to be available at runtime, and doing runtime checks is absolutely not an option.

Even if we could check the runtime image, I don't like this approach because it puts a lot of extra work on the buildpacks.

I can't speak for all of the Paketo buildpacks, but for the Java-related ones that supply binaries, we already kind of do option 0.) because we just have stack = "*" everywhere. It hasn't been much of an issue, but then again the binaries we use have very few shared library dependencies (usually just glibc).

There are a couple of places where we need to use different start commands based on if a shell is present, like Tomcat, and that's probably where we'd use the target id.

Thoughts on $CNB_TARGET_ID "namespacing"

100%

A note on ABI compatibility

100%. The only other way around is rebuilding stuff. That works for some build systems like Node.js/NPM that will recompile native code, but not for PHP extensions. Not sure there's a good way around it. I think the Paketo PHP buildpack is just more tightly coupled to where it can run than other buildpacks.

[dmikusa]

But it does give you information on what versions of libraries you can expect, if they are there. If you have "ubuntu" and "22.04" as target distribution name/version, then you know that libonig will be at ABI version 5, and you get version 6.9.7.1 or later, because that's what Ubuntu ships in jammy: https://packages.ubuntu.com/jammy/libonig5

I can't speak for others, but this is not the question I'm usually trying to answer in a buildpack.

What I usually need to know is if XYZ is actually present.

• XYZ might be a requirement of a binary being installed
• XYZ might be a build requirement of something that needs to compile
• XYZ might be a tool that needs to be run
• The presence of XYZ might impact how the buildpack operates. i.e. it emits a different start command if something is present, or compiles a binary in a different way if something is missing.

Buildtime is easy enough. It's more work, but we can check for the presence of things in the buildpack. Runtime, I just don't understand how we'd be able to provide any assurances about our builds.

What I foresee happening is a.) people using alt run images, the build succeeds, but the app doesn't start, so they come to complain to the buildpack author, and b.) because of a.) we end up having to have some policy like you use the paketo base images or we can't support what you're doing. I don't like it, but having the build pass and the app not start successfully is a big problem.

[dzuelke]

What I foresee happening is a.) people using alt run images, the build succeeds, but the app doesn't start, so they come to complain to the buildpack author, and b.) because of a.) we end up having to have some policy like you use the paketo base images or we can't support what you're doing. I don't like it, but having the build pass and the app not start successfully is a big problem.

But that can already happen today with stack IDs and people bringing their own run images, right?

[dmikusa]

Yes, presently I'd say we (Paketo Java team) are here. Swim at your own risk.

We technically allow * wildcard stack with Paketo Java buildpacks, so people can try it on other stacks, but we can't support their custom stacks. If someone came to me with a problem and was using a non-Paketo stack, I'd ask them to try the Paketo stack and if the problem doesn't occur there also they'd be on their own.

My concern is not that we're here presently, but that this new proposal won't put us in a better situation. We have limited dev cycles on the Paketo Java team, and plenty of work queued up. If this isn't going to offer us a better situation, why move ahead with this?

I hate to be that person that complains and doesn't offer some sort of solution, but idk exactly what would help here. We've tried adding more metadata about packages to the stack (i.e. mixins) and that didn't really work out well. It's also not dynamic enough in the context of extensions. The buildpack really needs to be able to introspect the actual runtime image or be provided with a reliable set of data about what packages are installed. If we want to improve the situation for buildpack authors. Otherwise, for me a buildpack author this will be a vertical move, no better but not wose.

[natalieparellano]

Even if we could check the runtime image, I don't like this approach because it puts a lot of extra work on the buildpacks.

This is fair. Though perhaps there are ways we could streamline the process?

Runtime, I just don't understand how we'd be able to provide any assurances about our builds.

You could in theory have image extensions that "extend" the run image simply by inspecting the packages that are installed and failing the build in the case that needed packages are missing. This could be circumvented by checks against known targets, etc.

[natalieparellano]

If this isn't going to offer us a better situation, why move ahead with this?

Allowing buildpacks to express the os/arch/etc they are expected to work with, and providing os/arch/etc information about the target image will move us one step forward along the path of multi-arch / cross-platform builds.

[jama22]

I believe we (Google Cloud's buildpacks) would face a lot of similar issues as the Heroku in this situation. For context, we maintain our own stacks to support the different ways customers want to interact with GCP products:

• google.gae.22 comes with lots of additional system packages to help developers get immediate access to libraries (e.g. ffmepg, imagemagick, etc.) so everything "just works"
• google.22 comes with "just the basics" for users who don't want to see those additional packages
• google.min.22 comes with the absolute minimum for the security oriented user

Package differences

A few thoughts/questions also come to mind:

• Overall I think it is a good thing that we're standardizing on OS distributions for compatibility. This will help answer a lot of developer confusion about the interoperability of buildpacks across platforms and providers
• Are we standardizing what it means to be on specific OS targets? If I target Ubuntu 22.04, is that equivalent to always using ubuntu:jammy? This might be helpful to reduce the potential difference between two builders that say they are on ubuntu 22.04, but have slightly different ways of sourcing their base images (update: @dmikusa describes that quite well above)
• Some of the issues raised will become serious barriers to adopting the Stacks-less world.
  • For authors, there is going to be a (short term) engineering burden to move to a stackless world and do some basic testing to make it work. In scenarios where the incompatabilities with other builders is just too annoying to overcome, I can see a lot of authors just giving up and saying "we only support distribution=google" and calling it a day. To be clear, this is something I'd want to avoid. This only works if we all choose to move to this standard in unison.
  • For consumers of buildpacks they want everything to "just work". Most of our customers don't even know/care that they're using buildpacks. We want to avoid the situation where we announce interoperability with the different buildpack providers only for them to find out that 90% of them don't work! This could be really detrimental to the brand of CNBs and stifle adoption even further.
  • For platform operators (e.g. Heroku Eng team, Google Eng team, Bloomberg Eng team, etc.) they need to know for absolute certainty that a buildpack that they're introducing to the system is going to work with their user base. I don't think any of us want to introduce breaking changes that will have an impact to the productivity of our consumers or the reliability of the platform

Package Management for Authors

In the future, CNBs will therefore have to handle the possibility of missing libraries for their pre-built language runtimes and other packages, as they may be executed in a builder that uses base images different from those provided by the buildpack vendor

Let's assume we can all come to a shared understanding on what it means to be on Ubuntu 22.04, and that it's always going to be the jammy tag published by Canonical on DockerHub.

Each buildpack author then could determine the additional packages that they require on top of Ubuntu 22.04. This is just a rough idea, but what if we let authors declare additional packages:

[[targets]]
os = "linux"
arch = "amd64"
[[targets.distributions]]
name = "ubuntu"
versions = ["22.04"]

# packages that should exist in the build image
[[targets.build]]
packages = ["libc-client.so.2007e", ...] #maybe could be purls instead of names?

# packages that should exist in the run image
[[targets.run]]
packages = ["libc-client.so.2007e", ...]

And then we'd extend the...Detect phase? to install the listed packeges for the selected buildpacks.

I think this could be interesting because:

• It would lower the stress/anxiety of integration testing with every buildpack provider to make sure their buildpack will work
• (potentially) maintains the rebaseable seams of the base image
• Outside of buildpack portability, it could actually help us create smaller build and run images. We think a lot about how to create "right-sized" images for our customers. We could start with a distroless / ubuntu-min image, and use additional buildpacks to dictate what packages to include to the build/run images.

Downsides:

• Not sure if this approach is possible, so sorry in advance if this is completely off the wall crazy
• This could drastically include build times if there's a lot of buildpacks with their own packages
• Creates all kinds of new problems for dependency management

[jabrown85]

The project had a similar concept a few years back (buildpacks/rfcs#40 and then buildpacks/rfcs#172)

https://github.com/buildpacks/rfcs/pull/172/files#diff-0938f247dbc403643bde3121747060d3f22df97dbcd2375ef2dce150471b01f6R188 even mentions using purl identifiers as an alternative for buildpack authors

It gets really complicated to express what may be needed.

• The buildpack itself may need libraries to even function on any given jammy build stack. The most common example I can think of is a buildpack written in bash that requires jq. The thing it is doing may not need jq, but the buildpack itself requires it to do the job.
• The app image being built may need postgresql client because the buildpack sees a postgresql gem in the Gemfile, but otherwise doesn't need that library if the app swapped to using mysql. Statically requiring postgresql libraries in this case at the buildpack.toml level would be mean a perfectly viable jammy builder may not be usable even if the app will never use postgresql.

[jama22]

Thanks for the thoughtful responses ya'll. @dzuelke the breakdown of how target ID would be used in our case helped me understand this design a lot better

[natalieparellano]

So, a new builder would have "rhel" and "9" as distribution name and version, but still com.heroku.os-images.standard as the target ID - the Ubuntu and RHEL images would have very similar packages.

Thank you, this helped me understand the concern about "target ID" not being a unique identifier.

[natalieparellano]

We want to avoid the situation where we announce interoperability with the different buildpack providers only for them to find out that 90% of them don't work!

100%!

[natalieparellano]

You can presently use them with build images, but not run images. i.e. it's not production ready now.

Lifecycle 0.17.0 (if we ever ship it 😵‍💫 ) will have support for run image extension. Though it's true image extensions as a whole will probably remain experimental until we have more feedback.

[schneems]

To add: With the removal of that stack id then some buildpack authors wouldn't be able to get far enough to fail on binary execution as they don't even know what binary to try to download. Currently, some Heroku buildpacks treat the stack id as a cache key of-sorts, so there is a <s3url>/heroku-22/ruby-3.1.2.tar.gz and <s3url>/heroku-20/ruby-3.1.2.tar.gz. With stacks removed, we'll need to either find some other key or do translation (for the Ruby buildpack where we target to compile for N different stacks).

I could convert Ubuntu 22.04 into heroku-22 but if someone forked ubuntu and had an identical OS . One option could be to try to use the latest and emit a warning that it might be unsupported. I would be curious how others are planning on managing this in their buildpacks

[dzuelke]

With stack ID removed, Heroku buildpacks will no longer pull from <s3url>/heroku-22/…, but from something like <s3url>/linux-amd64/ubuntu-22.04/…. See the example curl under option 2 in the original post at the top.

The Heroku Ruby buildpack will then most likely also work on https://github.com/GoogleCloudPlatform/buildpacks/tree/main/stacks/google_gae_22 and https://github.com/paketo-buildpacks/builder-jammy-full

Assuming of course all the libraries the Heroku build of Ruby linked against are on those build and run images. Helping ensure that is... why I started this discussion ;)

[dzuelke]

Responding mostly inline, but I realize I had a typo in an important paragraph. It is not possible to match $CNB_TARGET_ID against [[target.distributions]], and that's how it should be.

Updated and slightly re-phrased:

While there may be a $CNB_TARGET_ID environment variable present during the detect and build phases, it is not (and should not be) possible to define this ID as part of the [[targets]] list in a buildpack.toml for the purpose of "matching" buildpacks against "stacks".

The resulting "looser" contract makes it easier to combine buildpacks that currently explicitly depend on different stacks - instead, they will declare compatibility with certain OS distributions and versions.

[dzuelke]

So, one thing that I did not touch on at all in the original post is alignment (or lack thereof) between build and run images.

A buildpack can only analyze the build image; the presence of a library during bin/build is no guarantee that it'll also exist in the run image.

I suppose the general assumption for "curated" images like Paketo's, Heroku's, Google's, is that the run images are a complete subset of the build images, but this may not always hold true, and certainly cannot be guaranteed once users have their own builders and images.

An evolution of "the packages don't match" is the "the distributions don't match" case (or even OS/architecture). Then we're effectively in "cross-building" territory.

The question is how realistic is this in the real world. A JVM app built on a Ubuntu builder, then executed on a minimal RHEL runtime image, sure - linked libc version in the JVM allowing. Same for e.g. Go apps with static linking, or some elaborate CGO_LDFLAGS gymnastics.

But a Ruby binary for a Ubuntu 22.04 runtime target is unlikely to work on an RHEL 7 builder. Too many differences in the dynamically loaded libraries for bundle install or maybe even ruby -v to succeed.

Those are likely use cases for people who are then bringing their own images anyway and have full end-to-end control over builders and build/run images.

But it does beg the question... is $CNB_TARGET_ID really about the target, and is it really an ID?

In fact... where is the value even really supposed to come from? https://github.com/buildpacks/spec/blob/main/buildpack.md#targets doesn't say that at the moment (I know... it's a work in progress 😄).

[edmorley]

I think one potentially common (for advanced users) case where the run image is not a subset of the build image, is say Go apps wanting a distroless run image.

[jabrown85]

Agree @edmorley - the paketobuildpacks/builder-jammy-tiny uses a distroless-like run image for a practical example

[natalieparellano]

is $CNB_TARGET_ID really about the target

That's how it was intended, though in practice some buildpacks will probably assume if the target ID is XXX then the builder must by YYY. For me, the value of this variable is really about A buildpack can only analyze the build image; the presence of a library during bin/build is no guarantee that it'll also exist in the run image. and If $CNB_TARGET_ID matches a specific known/expected value, a buildpack may skip this step. See notes on $CNB_TARGET_ID namespacing below.

In fact... where is the value even really supposed to come from?

Cross-posting from Slack, in case others have this question:

Run image io.buildpacks.base.id -> CNB_TARGET_ID
Run image config -> CNB_TARGET_OS
Run image config -> CNB_TARGET_ARCH
Run image config -> CNB_TARGET_ARCH_VARIANT (this is why I don’t like CNB_TARGET_VARIANT as a possible alternative to CNB_TARGET_ID !)
Run image io.buildpacks.base.distro.name -> CNB_TARGET_DISTRO_NAME
Run image io.buildpacks.base.distro.version -> CNB_TARGET_DISTRO_VERSION

[dzuelke]

Okay, so, thinking about what $CNB_TARGET_ID really is useful for in combination with the distribution name and version... it's declaring compatibility, right?

In its simplest form, it expresses (or promises, really) a run images' compatibility with the list of packages that are declared somewhere (or that the image has always offered, since the spec says "MUST NOT break ABI compatibility" for newer image versions).

In Paketo's builder, imagine this run-image metadata:

[run-image]
   image = "…"
   reference = "…"
   [target]
   id = "io.paketo.run.full"
   os = "linux"
   arch = "amd64"
   [target.distro]
   name = "ubuntu"
   version = "22.04"

And in Heroku's builder, this:

[run-image]
   image = "…"
   reference = "…"
   [target]
   id = "com.heroku.runtime.standard"
   os = "linux"
   arch = "amd64"
   [target.distro]
   name = "ubuntu"
   version = "22.04"

A buildpack that downloads pre-built binaries such as language runtimes (with a bunch of libraries dynamically linked) declares compatibility with Ubuntu 22.04, since that what it has pre-built binaries for:

api = "0.10"
[buildpack]
id = "…"
name = "…"
version = "…"

[[order]]
[[order.group]]
# …

[[targets]]
os = "linux"
arch = "amd64"
[[targets.distros]]
name = "ubuntu"
version = "22.04"

This buildpack therefore would work on both of the builders above, but to be reasonably confident that, at runtime, the libraries its binaries dynamically link against exist, it now checks against some known run images it supports:

# bin/compile

# ... download binary using $CNB_TARGET_OS, `$CNB_TARGET_DISTRO_NAME`, etc.

if os.getenv("CNB_TARGET_ID") not in ["io.paketo.run.full", "com.heroku.runtime.standard", "com.google.gcp.gae"]:
    # ... do an LDD check even if that's no guarantee if the run image mismatches
    # ... or print a warning, at least in case something fails later, about required libraries

So far, so good.

But.

---

This would cause lots of work, for everyone, long-term

What if...

1. ... somebody uses a builder with their own custom images based on, say, "io.paketo.run.full"?
   • how can they express that the image contents are equivalent to "io.paketo.run.full", plus just a few custom packages added?
   • ... and do this in a way that allows existing buildpacks that are written with popular builders/images in mind (see Python code sample above) to detect this equivalence?
2. ... Heroku want to split their single-sized "standard" stack into "com.heroku.runtime.minimal", "com.heroku.runtime.standard", and "com.heroku.runtime.supersized" variants...
   • how can countless existing buildpacks in the wild know that "supersized" is a superset of "standard" and detect it correctly?
   • would it take a lot of PRs by Heroku engineers to a lot of buildpacks' codebases?
3. ... a buildpacks' binaries really only need "com.google.gcp.minimal"; how can this buildpack's authors avoid
   • ... having to also specify the equally compatible "com.google.gcp.basic" and "com.google.gcp.gae" (as they're both supersets) in their code?
   • ... having to add potentially any future new image variants to their code?

---

Make it not a single ID, but a set of labels!

Think of it instead this way: we want compatibility "labels", and we want to be able to express more than just a single one for an image.

Let's call the resulting env var, just for the sake of this example, $CNB_TARGET_COMPATIBILITY.

And let's define this env var as a comma-separated list of the compatibility labels (spec'd as a set, really, since the order is irrelevant), which keeps things super simple, because on the image side, it always means "a union of those labels", while buildpacks, during execution, can match against it with arbitrary logic.

So, our images from earlier, would now have metadata like this:

[run-image]
   # …
   [target]
   compatibility = ["io.paketo.run.tiny", "io.paketo.run.full"]
   os = "linux"
   arch = "amd64"
   [target.distro]
   name = "ubuntu"
   version = "22.04"

[run-image]
   # …
   [target]
   compatibility = ["com.heroku.runtime.standard"]
   os = "linux"
   arch = "amd64"
   [target.distro]
   name = "ubuntu"
   version = "22.04"

And a buildpack can parse the env var:

# bin/compile

# ... download binary using $CNB_TARGET_OS, `$CNB_TARGET_DISTRO_NAME`, etc.

compatible_targets = set(os.getenv("CNB_TARGET_COMPATIBILITY").split(","))
tested_targets = {"io.paketo.run.full", "com.heroku.runtime.standard", "com.google.gcp.gae"}
# check if comma-separated CNB_TARGET_COMPATIBILITY set intersects with one of our tested-against run images
if not tested_targets & compatible_targets:
    # ... do an LDD check even if that's no guarantee if the run image mismatches
    # ... or print a warning, at least in case something fails later, about required libraries

---

Custom images are suddenly no problem for buildpacks!

I can now make my own image for my own builder, where I use a mix of Paketo and Google buildpacks, so I ensured that the run image is a union of them, because that's what I decided is best for my purposes (I am not giving it its own "compatibility name" here, since I do not plan on sharing it with third parties, but... I could):

[run-image]
   # …
   [target]
   compatibility = ["io.paketo.run.tiny", "io.paketo.run.full", "com.google.gcp.minimal", "com.google.gcp.basic", "com.google.gcp.gae"]
   os = "linux"
   arch = "amd64"
   [target.distro]
   name = "ubuntu"
   version = "22.04"

Any buildpack that has logic specifically for any of those labels will continue to "just work" as if the image wasn't custom.

Image "flavors" are suddenly trivial to evolve!

At Heroku, let's say we decide to start offering minimal images. No problem; the existing "standard" image's metadata changes accordingly:

[run-image]
   # …
   [target]
   compatibility = ["com.heroku.runtime.minimal", "com.heroku.runtime.standard"]
   os = "linux"
   arch = "amd64"
   [target.distro]
   name = "ubuntu"
   version = "22.04"

A buildpack currently somehow special-cases "com.heroku.runtime.standard"? Great, that still works, but new buildpacks can start "targeting" the com.heroku.runtime.minimal variant, and they'll work on both, as well as a potential future com.heroku.runtime.supersized variant.

And buildpack authors do not have to lift a finger!

The beautiful thing is that no existing buildpack that has explicit "shortcuts" or automated tests for well-known run images need any adjustments when new image variants are introduced that are a superset of others.

This scales really well as the number of builders, images and buildpacks goes up over time.

---

More use cases

Easy images for debugging

Another cool thing that becomes possible are debug run images. What if our website keeps crashing server-side with a segfault in libXML? Well...:

[run-image]
   # …
   [target]
   compatibility = ["io.paketo.run.tiny", "io.paketo.run.full", "io.paketo.run.debug"]
   os = "linux"
   arch = "amd64"
   [target.distro]
   name = "ubuntu"
   version = "22.04"

The extra entry in compatibility isn't even necessary for this to work, but... such an image would have https://wiki.ubuntu.com/Debug%20Symbol%20Packages for all libraries, plus some useful stuff like dbg and valgrind.

Because it still identifies as compatible with "io.paketo.run.full" (and "io.paketo.run.tiny"), buildpacks targeting this image still behave the same - although they could use the presence of such a well-known indicator to, in turn, download debug symbols for their own binaries!

"Fake It 'Till You Make It"

Or, what if CNBs and the infrastructure become the greatest business on the planet, and we have wonderful competition between vendors.

A new company enters the marketplace and promises the very best builder images.

How can they ever catch up to the big, established players, whose $CNB_TARGET_COMPATIBILITY entries are used by many buildpacks in the wild to short-circuit sanity checks, and in automated testing?

Well, this new company fully controls their own builder and images, including the compatibility field in the image metadata... ;)

Really weird buildpack requirements

This is quite unlikely to be a real use case, but the buildpack code that matches against $CNB_TARGET_COMPATIBILITY could, of course, decide that it needed several compatibilities at once, for whatever reason.

Maybe a super rare library is needed for a binary, and it's known to work on

1. com.google.gcp.gae, or on
2. the union of io.paketo.run.full and com.heroku.runtime.standard.

The latter would of course require users of that buildpack to then have a custom image with the right packages, and both of those compatibility labels in its compatibility metadata field.

---

What's needed to make this happen?

1. re-define the various related ".id" fields to ".compatibility" and make it a set/list/array, not a single string;
2. re-define $CNB_TARGET_ID to $CNB_TARGET_COMPATIBILITY, serialized as a comma-separated string of the above values;
3. optionally mention in the spec that no matching ever occurs during detection etc on these labels (this is implicit already);
4. optionally mention in the spec that buildpacks should not fail on unknown or missing compatibility label values.