Application image labels too long for containerd

[jimmykarily]

What happened?

I'm building an image using the full-builder. I'm doing this through Epinio. The cluster I'm using is k3d using the image rancher/k3s:v1.22.2-k3s2. That image is running containerd v1.5.7 in it.

When containerd tries to start a container using the paketobuildpacks/builder:full image, it fails with an error:

label key and value greater than maximum size (4096 bytes), key: io.buildpa: invalid argument 

This seems to originate here: https://github.com/jessvalarezo/containerd/blob/18c4322bb3ddcb8f8b4eea2f3c027a06194041b4/labels/validate.go

Everything works fine on containerd v1.4.11. I had a look at changelogs and I think it's this one (introduced in 1.5.6):

containerd/containerd#5938
https://github.com/containerd/containerd/pull/6012/files

Is there any reason we we have such long labels on the image?
There have been similar issues in the past: #191

• What were you attempting to do?

Create a container from paketobuildpacks/builder:full with containerd 1.5.6

• What did you expect to happen?

the container to be started

• What was the actual behavior? Please provide log output, if possible.

See description. Original issue started here: epinio/epinio#833 (comment)

Build Configuration

• What platform (pack, kpack, tekton buildpacks plugin, etc.) are you
  using? Please include a version.

Epinio build from here: epinio/epinio#868

• What buildpacks are you using? Please include versions.

Didn't get that far

• What builder are you using? If custom, can you provide the output from pack inspect-builder <builder>?

paketobuildpacks/builder:full (sha256:e68c940932d2cf17ec10dbc9a4de7af08deec728d7cf5de9278803ec0f96ba6e)

Checklist

• I have included log output.
• The log output includes an error message.
• I have included steps for reproduction. (kinda)

[fg-j]

Hi @jimmykarily. Currently, Paketo creates builder images using the pack builder create command (with the pack CLI). When pack creates the images, it adds a lot of metadata as a JSON object stored under theio.buildpacks.builder.metadata key. This metadata describes the contents of the builder (e.g. which buildpacks it contains). The Paketo Full builder contains the most buildpacks of any of our offerings, which is likely why you're running into this limitation.

Since pack is maintained by the Cloud Native Buildpacks (CNB) project, it's not directly in our control to reduce what's in the builder labels, but I've raised this in the CNB slack. (Here's the thread.)

In the meantime, a workaround might be to use the Paketo Buildpackless Full Builder. It provides the same build and run-time base images as the Paketo Full Builder, but contains no buildpacks, so the builder image metadata is much shorter. You'd need to explicitly specify buildpacks at build time to use the builder.

[jimmykarily]

@fg-j thank you for you answer. I opened an issue on the pack repo. Should we close this one or should we keep open until pack has a fix and it's bumped in the full builder image?

[fg-j]

Let's keep this open for now. It'll help remind us to keep an eye on the various related upstream things.

[ForestEckhardt]

@jimmykarily Hey there, just doing a quick bump. Is this still and issue that you are running into?

[jimmykarily]

@ForestEckhardt it's fixed. I'm closing this, thanks.