failed to load LLB: runtime execution on platform linux/arm64 not supported

[tamalsaha]

We started using the new docker buildx to build arm64 images. But it is not working with debian base image.

DOCKER_CLI_EXPERIMENTAL=enabled docker buildx build --platform linux/arm64 --load --pull -t kubedb/operator:v0.13.0-rc.0-dbg_linux_arm64 -f bin/.dockerfile .

container: kubedb/operator:v0.13.0-rc.0-dbg_linux_arm64
[+] Building 1.9s (3/3) FINISHED                                                                                                                                                                            
 => [internal] load build definition from .dockerfile-DBG-linux_arm64                                                                                                                                  0.5s
 => => transferring dockerfile: 652B                                                                                                                                                                   0.0s
 => [internal] load .dockerignore                                                                                                                                                                      0.3s
 => => transferring context: 2B                                                                                                                                                                        0.0s
 => [internal] load metadata for docker.io/library/debian:stretch                                                                                                                                      1.2s
failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to load LLB: runtime execution on platform linux/arm64 not supported
Makefile:205: recipe for target 'bin/.container-kubedb_operator-v0.13.0-rc.0_linux_arm64-DBG' failed
make[2]: *** [bin/.container-kubedb_operator-v0.13.0-rc.0_linux_arm64-DBG] Error 1
Makefile:105: recipe for target 'push-linux_arm64' failed
make[1]: *** [push-linux_arm64] Error 2
Makefile:307: recipe for target 'release' failed
make: *** [release] Error 2

Here is the docker file:

FROM debian:stretch

ENV DEBIAN_FRONTEND noninteractive
ENV DEBCONF_NONINTERACTIVE_SEEN true

RUN set -x \
  && apt-get update \
  && apt-get install -y --no-install-recommends apt-transport-https ca-certificates tzdata locales openssl \
  && rm -rf /var/lib/apt/lists/* /usr/share/doc /usr/share/man /tmp/* \
  && localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8 \
  && echo 'Etc/UTC' > /etc/timezone && dpkg-reconfigure tzdata

ENV TZ     :/etc/localtime
ENV LANG   en_US.utf8
ENV LC_ALL en_US.UTF-8

ADD bin/linux_arm64/operator /operator

ENTRYPOINT ["/operator"]

I am getting anything with google search for failed to solve with frontend dockerfile.v0: failed to load LLB: runtime execution on platform linux/arm64 not supported.

On the same machine, the following arm64 image is building ok.

FROM gcr.io/distroless/base

ADD bin/linux_arm64/operator /operator

# This would be nicer as `nobody:nobody` but distroless has no such entries.
USER 65535:65535

ENTRYPOINT ["/operator"]

Any help is appreciated.

[tamalsaha]

I am running this on Ubuntu 18.04 and following https://community.arm.com/developer/tools-software/tools/b/tools-software-ides-blog/posts/getting-started-with-docker-for-arm-on-linux

$ uname -a
Linux beast 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

$ docker info
Client:
 Debug Mode: false
 Plugins:
  app: Docker Application (Docker Inc., v0.8.0)
  buildx: Build with BuildKit (Docker Inc., v0.2.2-10-g3f18b65-tp-docker)

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 105
 Server Version: 19.03.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 4.15.0-58-generic
 Operating System: Ubuntu 18.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.38GiB
 Name: beast
 ID: H747:N42V:6KTW:VQWA:RUJL:6U73:XTFK:VSPF:GWTL:KHMA:POO7:O2AN
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: tigerworks
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

[tonistiigi]

you need to install qemu-user #132 (comment)

[tamalsaha]

Thanks @tonistiigi . It seems that I had already installed it but still getting this error.

$ sudo apt-get install qemu-user-static
[sudo] password for tamal: 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
qemu-user-static is already the newest version (1:2.11+dfsg-1ubuntu7.17).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

$ env DOCKER_CLI_EXPERIMENTAL=enabled docker buildx build --platform linux/arm64 --load --pull -t tigerworks/operator:fdocker .
[+] Building 1.5s (3/3) FINISHED                                                                                                                                                                            
 => [internal] load .dockerignore                                                                                                                                                                      0.4s
 => => transferring context: 2B                                                                                                                                                                        0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                                   0.6s
 => => transferring dockerfile: 32B                                                                                                                                                                    0.0s
 => [internal] load metadata for docker.io/library/debian:stretch                                                                                                                                      0.7s
failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to load LLB: runtime execution on platform linux/arm64 not supported

[tamalsaha]

I had a mybuilder that was not working.

$ env DOCKER_CLI_EXPERIMENTAL=enabled docker buildx ls
NAME/NODE    DRIVER/ENDPOINT             STATUS  PLATFORMS
mybuilder *  docker-container                    
  mybuilder0 unix:///var/run/docker.sock running linux/amd64, linux/386
default      docker                              
  default    default                     running linux/amd64

I rm-rf the old builder and then created a new one and things are working now.

$ docker buildx create --name nubuilder
nubuilder
$ docker buildx use nubuilder
$ docker buildx inspect --bootstrap
[+] Building 9.6s (1/1) FINISHED                                                                                                                                                                            
 => [internal] booting buildkit                                                                                                                                                                        9.6s
 => => pulling image moby/buildkit:master                                                                                                                                                              5.8s
 => => creating container buildx_buildkit_nubuilder0                                                                                                                                                   3.8s
Name:   nubuilder
Driver: docker-container

Nodes:
Name:      nubuilder0
Endpoint:  unix:///var/run/docker.sock
Status:    running
Platforms: linux/amd64, linux/arm64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

So, the problem was the old builder did not have arm64. I don't know why. Thanks @carlosedp helping me get this working.

[tonistiigi]

You need to restart or recreate the builder for it to pick up new qemu support.

[ghost]

Hi, have the same problem, am working with rhel7 and runed'install qemu-user', but when run 'buildx ls', it show only 'running linux/amd64, linux/386', and it don't show 'linux/amd64, linux/arm64...'. What can I have do, please?

[tamalsaha]

Delete the old builder and create a new builder.

…

On Fri, Aug 30, 2019, 5:50 PM Evandro Nascimento ***@***.***> wrote: Hi, have the same problem, am working with rhel7 and runed'install qemu-user', but when run 'buildx ls', it show only 'running linux/amd64, linux/386', and it don't show 'linux/amd64, linux/arm64...'. What can I have do, please? — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#138?email_source=notifications&email_token=AAAXEXQ6LNPMVBR3CWTPPITQHGIZRA5CNFSM4IOX6CDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5S3WEQ#issuecomment-526760722>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAXEXUA4XDHSWIJAAVWAT3QHGIZRANCNFSM4IOX6CDA> .

[ghost]

Sorry, i am begin work with this now, how can I delete the old builder e create the new?

[tamalsaha]

docker buildx rm $name

…

On Fri, Aug 30, 2019, 6:01 PM Evandro Nascimento ***@***.***> wrote: Sorry, i am begin work with this now, how can I delete the old builder e create the new? — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#138?email_source=notifications&email_token=AAAXEXXBHO7V6CKOYWET4ETQHGKB5A5CNFSM4IOX6CDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5S4H7Q#issuecomment-526763006>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAXEXVCIFVMMDFR7UJOJQLQHGKB5ANCNFSM4IOX6CDA> .

[ghost]

ok, thanks a lot! I will do this.

[ghost]

I see I need to read a little more about it, but I'm having a little trouble finding something about using this buildx technology ... I'd like to create a raspbian / arm docker image in my rhel7 environment, but I'm with some difficulty. Could you please pass some link with material that could help me. I'll be very grateful. Thank you!

[tamalsaha]

#138 (comment)

…

On Fri, Aug 30, 2019, 6:16 PM Evandro Nascimento ***@***.***> wrote: I see I need to read a little more about it, but I'm having a little trouble finding something about using this buildx technology ... I'd like to create a raspbian / arm docker image in my rhel7 environment, but I'm with some difficulty. Could you please pass some link with material that could help me. I'll be very grateful. Thank you! — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#138?email_source=notifications&email_token=AAAXEXUR2QUOVCY5L5RH2VLQHGL2NA5CNFSM4IOX6CDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5S473I#issuecomment-526766061>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAXEXRPOKJRIJ4I7EP7JATQHGL2NANCNFSM4IOX6CDA> .

[dzvid]

It worked for me after installing qemu-user and running the getting started tutorial at multiarch/qemu-user-static

[wohali]

I found that on my work image, I have to constantly re-run the multiarch/qemu-user-static process after each boot. It's a simple enough workaround for us anyway.

[hfedcba]

For me running

docker run --rm --privileged docker/binfmt:66f9012c56a8316f9244ffd7622d7c21c1f6f28d

solved the problem.

[HeroCC]

Sorry to bug anyone, but I'm having the same issue. I've been trying to get this to work for a while now but can't seem to do it. I run the following, and yet my buildx builder doesn't have the new platforms supported, and the build gets that same error. The base image I'm trying to build is ubuntu:18.04 and I'm running on GitlabCI shared infrastructure (docker:stable image, dind service running, privileged containers enabled).

docker run --rm --privileged multiarch/qemu-user-static:register --reset
docker buildx create --use
docker buildx inspect --bootstrap
  * Platforms: linux/amd64, linux/386
docker buildx build --load --platform "linux/amd64,linux/arm64,linux/arm/v7" -t name .

Full Build Logs: https://gitlab.com/moos-ivp/moosdocker/-/jobs/396347211
CI Configuration / scripts: https://gitlab.com/moos-ivp/moosdocker/blob/d8bc10615ddf928c8fda8cf250cfa2c973734e29/.gitlab-ci.yml

Thank you!

[Silex]

@HeroCC: try docker run --rm --privileged multiarch/qemu-user-static --reset -p yes instead and it should work

Note that docker buildx ls will not list linux/arm/v7 as platform but it works.

[joariasl]

If you are using buildx with a Docker In Docker instance, you must be sure that the buildx instance created and the configuration of usage is made from the specific user of the runner. Check it with the docker buildx inspect command and be sure that the .docker/buildx folder exists.

[brainstorm]

@Silex, I tried docker run --rm --privileged multiarch/qemu-user-static --reset -p yes on a RHEL7 machine, but all I'm getting in both regular user and root is:

Setting /usr/bin/qemu-aarch64-static as binfmt interpreter for aarch64
sh: write error: Invalid argument
Setting /usr/bin/qemu-aarch64_be-static as binfmt interpreter for aarch64_be
sh: write error: Invalid argument
Setting /usr/bin/qemu-hppa-static as binfmt interpreter for hppa
sh: write error: Invalid argument
Setting /usr/bin/qemu-riscv32-static as binfmt interpreter for riscv32
sh: write error: Invalid argument
Setting /usr/bin/qemu-riscv64-static as binfmt interpreter for riscv64
sh: write error: Invalid argument
Setting /usr/bin/qemu-xtensa-static as binfmt interpreter for xtensa
sh: write error: Invalid argument

I guess that the problem comes from the fact that I have qemu-user instead of qemu-user-static installed on my system... does it really have to be the static one? Do you have/know of any other container that works with qemu-user instead?:

$ yum search qemu
Loaded plugins: enabled_repos_upload, package_upload, product-id, search-disabled-repos, subscription-manager
Repo rhel-7-server-extras-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/4119786289710741335-key.pem
Repo rhel-7-server-satellite-tools-6.5-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/4119786289710741335-key.pem
Repo rhel-7-server-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/4119786289710741335-key.pem
============================================================================================== N/S matched: qemu ===============================================================================================
ipxe-roms-qemu.noarch : Network boot loader roms supported by QEMU, .rom format
libvirt-daemon-driver-qemu.x86_64 : QEMU driver plugin for the libvirtd daemon
qemu.x86_64 : QEMU is a FAST! processor emulator
qemu-common.x86_64 : QEMU common files needed by all QEMU targets
qemu-guest-agent.x86_64 : QEMU guest agent
qemu-img.x86_64 : QEMU command line tool for manipulating disk images
qemu-kvm.x86_64 : QEMU is a machine emulator and virtualizer
qemu-kvm-common.x86_64 : QEMU common files needed by all QEMU targets
qemu-system-alpha.x86_64 : QEMU system emulator for Alpha
qemu-system-arm.x86_64 : QEMU system emulator for ARM
qemu-system-cris.x86_64 : QEMU system emulator for CRIS
qemu-system-lm32.x86_64 : QEMU system emulator for LatticeMico32
qemu-system-m68k.x86_64 : QEMU system emulator for ColdFire (m68k)
qemu-system-microblaze.x86_64 : QEMU system emulator for Microblaze
qemu-system-mips.x86_64 : QEMU system emulator for MIPS
qemu-system-moxie.x86_64 : QEMU system emulator for Moxie
qemu-system-or32.x86_64 : QEMU system emulator for OpenRisc32
qemu-system-s390x.x86_64 : QEMU system emulator for S390
qemu-system-sh4.x86_64 : QEMU system emulator for SH4
qemu-system-unicore32.x86_64 : QEMU system emulator for Unicore32
qemu-system-x86.x86_64 : QEMU system emulator for x86
qemu-system-xtensa.x86_64 : QEMU system emulator for Xtensa
qemu-user.x86_64 : QEMU user mode emulation of qemu targets
qemu-kvm-tools.x86_64 : KVM debugging and diagnostics tools

[Silex]

@brainstorm: see multiarch/qemu-user-static#38 and multiarch/qemu-user-static#100

[brainstorm]

Gotcha, thanks! That'll never work with a 3.10.0-693.2.2.el7.x86_64 kernel it seems, too legacy RHEL7 :/

[John-Dormevil]

Thanks it's works for me on azure pipelines:

• I installed qemu
• I removed and created a new builder

[zhangguanzhang]

$ docker buildx ls
NAME/NODE DRIVER/ENDPOINT             STATUS   PLATFORMS
test *    docker-container                     
  test0   unix:///var/run/docker.sock inactive linux/amd64, linux/arm64
default   docker                               
  default default                     running  linux/amd64, linux/386

# remove the buildx
$ docker buildx rm test

# enabled binfmt_misc
$ docker run --rm --privileged docker/binfmt:a7996909642ee92942dcd6cff44b9b95f08dad64

# create the builder
$ docker buildx create  --name test --platform linux/amd64,linux/arm64

# start the buildx
$ docker buildx inspect test --bootstrap

# set current buidler instance
$ docker buildx use test

# build the multi images
$ docker buildx build -t zhangguanzhang/keepalived:v2.0.20 . --push --platform linux/amd64

[rfay]

Unfortunately the requirement to run docker run --privileged means that even a docker buildx build can't be used in lots of environments.

[DHclly]

@brainstorm: see multiarch/qemu-user-static#38 and multiarch/qemu-user-static#100：参见 multiarch/qemu-user-static#38 和 multiarch/qemu-user-static#100

thank you very much