No support for CPU variant (like v7 for ARM) on customPlatform builds

[Silvanoc]

Actual behavior

The moment that --customPlatform=linux/arm gets specified, the variant v5 gets selected for those multi-arch images supporting linux/arm/v5 and linux/arm/v7. There's no possibility to specify architecture arm, variant v7 on customPlatform builds (I've also tried the specification that works for Docker linux/arm/v7). As a consequence, for example for Debian images only the older armel architecture can be selected, not the newer armhf.

Expected behavior
An option to specify the architecture variant (e.g. as mentioned linux/arm/v7) when more than one supported (like is the case for ARM32).

To Reproduce
Steps to reproduce the behavior:

1. Get a container host with binfmt support for arm/v5 and arm/v7.
2. Execute a Kaniko build using the below provided Dockerfile with the option --customPlatform=linux/arm.
3. Observe output armel (instead of the expected armhf).

Additional Information

• Dockerfile

FROM debian:stable

RUN dpkg --print-architecture

• Build Context
  No files needed
• Kaniko Image (fully qualified with digest)
  gcr.io/kaniko-project/executor:v1.5.1-debug, sha256:e00dfdd4a44097867c8ef671e5a7f3e31d94bd09406dbdfba8a13a63fc6b8060

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing	No new feature
Please check if the build works in docker but not in kaniko	works in docker with the option --platform linux/arm/v7
Please check if this error is seen when you use --cache flag	Yes, the error is still being seen
Please check if your dockerfile is a multistage dockerfile	No multistage

[tobias-schuele]

Any news on this? Any help would be highly appreciated.

[mickkael]

Hi,
this is where I have coded this option

kaniko/pkg/executor/build.go

Line 612 in 0477900

} else {

I'm not sure how to specify the v5/v7 from there

[mickkael]

Hum, I've digged a bit, and setting the variable GOARM to the value 5,6 or 7 would make it. I'll try to work on a PR for that

[mickkael]

This is actually what holds the configuration for the build of the base image
https://pkg.go.dev/github.com/google/go-containerregistry@v0.5.0/pkg/v1#ConfigFile
and in details:
https://github.com/opencontainers/image-spec/blob/master/config.md

There is nothing about GOARM...

digging further in the code of the OCI definition of a container, there is something I can try:
https://github.com/opencontainers/image-spec/blob/859973e32ccae7b7fc76b40b762c9fff6e912f9e/specs-go/v1/descriptor.go#L61

[mickkael]

hum. The build is failing. It seems like the variant option is still being discussed in the OCI definition, and not merged yet.
We will have to wait for the following merge before I can bring the feature.
opencontainers/image-spec#777

[Silvanoc]

Hi,
this is where I have coded this option

kaniko/pkg/executor/build.go

Line 612 in 0477900

} else {

I'm not sure how to specify the v5/v7 from there

In the case of Go cross-builds you need to provide the environment variable GOARM=7 in order to get binaries for ARMv7. I don't know if that helps @mickkael any further...

[Silvanoc]

@mickkael you're right, that the image-manifest is lacking a mechanism to differentiate ARM5 and ARM7. But the image-index does provide it and that should be all needed by Kaniko to get the right image (what I suppose is failing).

Missing the variant information in the image-manifest is bad for checking the architecture variant if the image-index is missing (what's the case of the Docker daemon) and that should get fixed in the specification. But thanks to the image-index it's not a big deal for getting the right arch variant from registries supporting image indexes (like the Docker Registry v2 does).

---

Just some commands to show what I mean:

Getting a Debian image for ARM5:

$ skopeo inspect --override-arch=arm --override-variant=v5 docker://docker.io/library/debian
...
    ],
    "Created": "2021-05-12T00:55:00.759065298Z",
...
    "Architecture": "arm",
    "Os": "linux",
    "Layers": [
        "sha256:db69af0c9ce605b61ac30d118d6c6ab4f8579b8c80e5469335f2108afa5d2c66"
    ],
...
}

Getting a Debian image for ARM5:

$ skopeo inspect --override-arch=arm --override-variant=v7 docker://docker.io/library/debian
...
    "Created": "2021-05-12T01:02:15.470295718Z",
...
    "Architecture": "arm",
    "Os": "linux",
    "Layers": [
        "sha256:89475607b1df9fc7eec7efe2fa845738a16cee3e92c1bb864c1f5a93b8303bc6"
    ],
...
}

Notice the different layer, although same build (only 7min difference in the timestamp). The confirmation comes when inspecting the image index:

$ skopeo inspect --raw docker://docker.io/library/debian | jq -r .
{
  "manifests": [
...
    {
      "digest": "sha256:5df4bb7023e48d9cbd8ad124853dee7b0c15f7b7fca3c7694601020e5af2b96f",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "arm",
        "os": "linux",
        "variant": "v5"
      },
      "size": 529
    },
    {
      "digest": "sha256:973ab50414f9597fdbd2b496e08eb22942722d9bb571c42e029c26229196259d",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "arm",
        "os": "linux",
        "variant": "v7"
      },
      "size": 529
    },
...
  "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
  "schemaVersion": 2
}

So the image index can help getting the right one, even though the image spec unfortunately doesn't report the right ARM version itself.

[Silvanoc]

I finally got to the conclusion that Kaniko currently has no support for CPU variants at all. This currently only affects ARM32, but might affect other platforms in the future, if the CPU variant feature gets new possible values (for ARM or for other architectures).

Current tries to fix it without expecting for the image-manifest in the OCI-image specification to get fixed are doomed to fail, because they try to put the CPU variant information into the structure hosting the image configuration (which isn't meant to have it as long as it's compatible with the current OCI-image specification). The key is that that information is not that critical if the right digest gets used to fetch the image. I haven't found out yet how to "teach" Kaniko how to get the right digest 🙁

[mickkael]

@Silvanoc , yes I went through the same. Let's track the PR in OCI.

[Silvanoc]

@mickkael I've just realized that supporting CPU variants implies two completely different things that can be tackled separately:

1. That the right base image gets pulled to build upon.
2. That the built image gets the CPU variant information.

The 2nd one cannot be fixed technically since the OCI-image specification doesn't support it. It isn't an issue in Kaniko and I agree that it has to be fixed somewhere else.

The 1st should be fixable though independently... that's in fact our requirement and the only one that can be placed on Kaniko as long as the OCI-image specification remains at its current version.

I'm the impression that I was focusing on the 1st one and you on the 2nd 🙂

[Silvanoc]

@mickkael I've realized that this comment of mine is completely useless, since that's what you were already mentioning in this comment of yours. Sorry, I oversaw it.

[mickkael]

No worries !

[mickkael]

@Silvanoc, the change is merged in the OCI. Maybe you should try to build again

[Silvanoc]

@mickkael great! Thanks for the heads-up!

Even with the OCI-image-spec fixing it, as long as Kaniko doesn't have support for CPU variants, it doesn't bring much. My MR adding support has been merged, but no release has been done yet providing it.