Cross-compiling `linux/amd64` on M1 mac #165

lpgauth · 2022-02-03T14:40:56Z

Hi, I'm trying to cross-compile an image using docker buildx and Colima but it seems to just hang on the build step...

How can I debug this? I looked at the logs but didn't see anything suspicious. The qemu VM seems idle....

docker buildx build --platform=linux/amd64 -t studio .
[+] Building 1236.6s (6/12)
 => [internal] load build definition from Dockerfile                                                                                                                                                             0.2s
 => => transferring dockerfile: 103B                                                                                                                                                                             0.2s
 => [internal] load .dockerignore                                                                                                                                                                                0.2s
 => => transferring context: 106B                                                                                                                                                                                0.2s
 => [internal] load metadata for docker.io/bitwalker/alpine-elixir-phoenix:latest                                                                                                                                0.4s
 => [internal] load build context                                                                                                                                                                                0.0s
 => => transferring context: 33.33kB                                                                                                                                                                             0.0s
 => CACHED [1/7] FROM docker.io/bitwalker/alpine-elixir-phoenix:latest@sha256:2c5773461305d76ebb32e23cbd1eeac718f3a1f2435d3b05dbae291f015fc0c3                                                                   0.0s
 => CANCELED [2/7] RUN mix do local.hex --force, local.rebar --force                                                                                                                                          1236.0s

colima version
colima version 0.3.2
git commit: 272db4732b90390232ed9bdba955877f46a50552

runtime: docker
arch: aarch64
client: v20.10.12
server: v20.10.11

The text was updated successfully, but these errors were encountered:

deviantintegral · 2022-02-05T03:13:38Z

I'm on Intel, but a few months ago I was cross-compiling for ARM without issue simply by passing --platform as above. Now, I can't run cross-architecture at all. It looks like over at #44 (comment) and https://github.com/abiosoft/colima/releases/tag/v0.3.0 things were changed so that now colima has separate VMs for each architecture.

Is there a reason to have separate VMs? For example, when testing Home Assistant builds, it will build containers all within one command for all of the different architectures.

abiosoft · 2022-02-05T04:16:13Z

Nothing changed, the separate VM was just an added option for anyone that prefers that.

@deviantintegral what is the output of docker buildx ls for you?

lpgauth · 2022-02-05T14:14:51Z

docker buildx ls
NAME/NODE DRIVER/ENDPOINT  STATUS  PLATFORMS
colima *  docker
  colima  colima           running linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/arm/v7, linux/arm/v6
default   docker-container
  default default          Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

deviantintegral · 2022-02-05T22:07:38Z

Looks like buildx is missing entirely. It sounds like buildx is included in deb and rpm packages, but not alpine? I'm also not seeing any qemu packages installed.

https://docs.docker.com/buildx/working-with-buildx/

docker/buildx#305 (comment)

docker buildx ls
docker: 'buildx' is not a docker command.
See 'docker --help'

I also used to be able to run cross-platform images, and not just through buildx:

❯ colima version
colima version HEAD-784cec0
git commit: 784cec04dc126b02f5d2697a864282d5d2301afe

runtime: docker
arch: x86_64
client: v20.10.12
server: v20.10.11
❯ colima status
INFO[0000] colima is running
INFO[0000] runtime: docker
INFO[0000] arch: x86_64
❯ docker run -it --rm --platform linux/aarch64 alpine:latest sh
standard_init_linux.go:228: exec user process caused: exec format error

It would be nice if colima could alert when there's a new VM image available. Should I file an issue for that?

Given how it seems you expected buildx to be present, I deleted the VM and recreated it. A new version was downloaded, and I'm able to run sh with aarch64. However, docker buildx ls still fails?

❯ docker run -it --rm --platform linux/aarch64 alpine:latest uname -a
Linux 0e20f7b7b87e 5.10.88-0-virt #1-Alpine SMP Tue, 28 Dec 2021 12:51:14 +0000 aarch64 Linux
❯ docker buildx ls
docker: 'buildx' is not a docker command.
See 'docker --help'

buildx isn't available inside the VM either:

❯ colima ssh
colima:/Users/andrew$ docker buildx
docker: 'buildx' is not a docker command.
See 'docker --help'
colima:/Users/andrew$

deviantintegral · 2022-02-05T22:10:54Z

It does look like docker build --platform=linux/arm64 works as expected.

abiosoft · 2022-02-06T08:04:28Z

Looks like buildx is missing entirely. It sounds like buildx is included in deb and rpm packages, but not alpine? I'm also not seeing any qemu packages installed.

It is not missing, it actually needs to be installed on same host as the docker client.

abiosoft · 2022-02-06T08:06:36Z

Hi, I'm trying to cross-compile an image using docker buildx and Colima but it seems to just hang on the build step...

How can I debug this? I looked at the logs but didn't see anything suspicious. The qemu VM seems idle....

@lpgauth is the Dockerfile something you can share? I can try building it on my end as well.

lpgauth · 2022-02-06T19:49:01Z

@abiosoft

git clone https://github.com/lpgauth/live_upload_example.git
cd live_upload_example
docker buildx build --platform=linux/amd64 -t test .

mritd · 2022-02-08T04:07:42Z

@lpgauth @abiosoft

I've made some discoveries, when I adjust the ulimit it seems to sometimes succeed and sometimes fail; on failure a core dump file is generated; I'm not familiar with erlang so I can't provide more information.

How to modify ulimit:

# install vim
colima ssh
sudo apk add vim

# change 'rc_ulimit' line to 'rc_ulimit="-c unlimited -n 65536 -i 65536 -s 65536 -l 65536 -u unlimited"'
vim /etc/init.d/docker
vim /etc/init.d/buildkitd

# restart colima
colima stop && colima start

lpgauth · 2022-02-08T13:37:25Z

@mritd interesting, I will try to replicate and have a look at the core dump.

nunovieira · 2023-04-11T22:50:52Z

Same problem. Frequent hangs during build or run with --platform linux/amd64.

-> % colima version
colima version 0.5.4
git commit: feef4176f56a7dea487d43689317a9d7fe9de27e

runtime: docker
arch: aarch64
client: v23.0.3
server: v20.10.20

-> % colima status
INFO[0000] colima is running using macOS Virtualization.Framework 
INFO[0000] arch: aarch64                                
INFO[0000] runtime: docker                              
INFO[0000] mountType: virtiofs                          
INFO[0000] socket: unix:///Users/xxx/.colima/default/docker.sock

-> % docker buildx ls
NAME/NODE     DRIVER/ENDPOINT STATUS  BUILDKIT PLATFORMS
colima *      docker                           
  colima      colima          running 20.10.20 linux/arm64, linux/amd64
default                       error            
desktop-linux                 error            

Cannot load builder default: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot load builder desktop-linux: Cannot connect to the Docker daemon at unix:///Users/xxx/.docker/run/docker.sock. Is the docker daemon running?

Is there anything I can do?

ArvinB · 2023-07-25T13:45:24Z

docker buildx ls
NAME/NODE DRIVER/ENDPOINT  STATUS  PLATFORMS
colima *  docker
  colima  colima           running linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/arm/v7, linux/arm/v6
default   docker-container
  default default          Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

@lpgauth Did you have to do something special to get all these different emulated architectures for colima? I am running on a Mac M1 Max, is there a limitation or workaround?

luisdavim · 2023-08-17T21:10:25Z

Same issue when compiling golang, I tried adjusting the ulimit but it didn't seem to make any difference.

luisdavim · 2023-08-17T23:00:15Z

starting colima with --vm-type vz --vz-rosetta does seem to help.

divStar · 2023-09-03T20:28:19Z

I think I have a similar problem. While I currently do not mind not being able to build for linux/amd64, it sure would be nice to be able to.

I never had Docker Desktop on this MacBook (M1, MacOS 13.5). I installed colima, docker, docker-compose and docker-buildx. The hello-world container works.

This is what docker buildx ls returns:

xx@~/some-project (master) docker buildx ls
NAME/NODE     DRIVER/ENDPOINT STATUS  BUILDKIT PLATFORMS
colima *      docker                           
  colima      colima          running 23.0.6   linux/arm64
default                       error            
desktop-linux                 error            

Cannot load builder default: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot load builder desktop-linux: Cannot connect to the Docker daemon at unix:///Users/igvo/.docker/run/docker.sock. Is the docker daemon running?

I am somehow missing linux/amd64 altogether.

This is my colima.yaml (I removed the comments for brevity):

cpu: 2

disk: 60

memory: 2

arch: aarch64

runtime: docker

kubernetes:
  enabled: false

  version: v1.27.1+k3s1

  k3sArgs:
    - --disable=traefik

autoActivate: true

network:
  address: false

  dns: []

  dnsHosts: {}

  driver: gvproxy

forwardAgent: false

docker: {}

vmType: vz

rosetta: true

mountType: virtiofs

mountInotify: true

cpuType: ""

layer: false

provision: []

sshConfig: true

mounts: []

env: {}

Can anyone see any reason why linux/amd64 is not listed in docker buildx ls?
~~And does anyone know how to get rid of the default and desktop-linux lines - since they apparently are not available?~~ Resolved it by creating a symlink for default (sudo ln /Users/xx/.colima/default/docker.sock /var/run/docker.sock) and removing desktop-linux (docker context rm desktop-linux).

ArvinB · 2023-11-21T16:16:16Z

@luisdavim

starting colima with --vm-type vz --vz-rosetta does seem to help.

I'm using the following to start colima...how would I incorporate the above?

brew services start colima

Information:

$ colima version
colima version 0.6.3
git commit: a9df8ba55e2ec9c5a9ecc4fb2ab941634b58ed3b

runtime: docker
arch: aarch64
client: v24.0.7
server: v24.0.5

$ docker buildx ls
NAME/NODE DRIVER/ENDPOINT STATUS  BUILDKIT             PLATFORMS
colima    docker                                       
  colima  colima          running v0.11.6+0a15675913b7 linux/arm64, linux/amd64, linux/amd64/v2

I am looking to use linux/s390x as an emulator.

Note: When doing a colima stop and colima start --vm-type vz --vz-rosetta I get the following:

FATA[0000] error starting vm: error at 'starting': exit status 1

Any hints or ideas anyone?

marcorichetta · 2024-05-13T20:06:44Z

Note: When doing a colima stop and colima start --vm-type vz --vz-rosetta I get the following:
FATA[0000] error starting vm: error at 'starting': exit status 1 
Any hints or ideas anyone?

For anyone who might drop into this issue, you can't change vm types on existent profiles.

jsoref mentioned this issue Mar 8, 2022

Support cross-arch containers rancher-sandbox/rancher-desktop#1764

Closed

1 task

This was referenced Sep 2, 2023

m1 mac x86_64 nodejs std::bad_alloc #723

Open

SEGFAULT building native gem extensions in v0.5.5 #794

Open

lpgauth closed this as completed Nov 28, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cross-compiling `linux/amd64` on M1 mac #165

Cross-compiling `linux/amd64` on M1 mac #165

lpgauth commented Feb 3, 2022 •

edited

Loading

deviantintegral commented Feb 5, 2022

abiosoft commented Feb 5, 2022 •

edited

Loading

lpgauth commented Feb 5, 2022

deviantintegral commented Feb 5, 2022

deviantintegral commented Feb 5, 2022

abiosoft commented Feb 6, 2022

abiosoft commented Feb 6, 2022

lpgauth commented Feb 6, 2022

mritd commented Feb 8, 2022

lpgauth commented Feb 8, 2022

nunovieira commented Apr 11, 2023 •

edited

Loading

ArvinB commented Jul 25, 2023

luisdavim commented Aug 17, 2023

luisdavim commented Aug 17, 2023

divStar commented Sep 3, 2023 •

edited

Loading

ArvinB commented Nov 21, 2023 •

edited

Loading

marcorichetta commented May 13, 2024

Cross-compiling linux/amd64 on M1 mac #165

Cross-compiling linux/amd64 on M1 mac #165

Comments

lpgauth commented Feb 3, 2022 • edited Loading

deviantintegral commented Feb 5, 2022

abiosoft commented Feb 5, 2022 • edited Loading

lpgauth commented Feb 5, 2022

deviantintegral commented Feb 5, 2022

deviantintegral commented Feb 5, 2022

abiosoft commented Feb 6, 2022

abiosoft commented Feb 6, 2022

lpgauth commented Feb 6, 2022

mritd commented Feb 8, 2022

lpgauth commented Feb 8, 2022

nunovieira commented Apr 11, 2023 • edited Loading

ArvinB commented Jul 25, 2023

luisdavim commented Aug 17, 2023

luisdavim commented Aug 17, 2023

divStar commented Sep 3, 2023 • edited Loading

ArvinB commented Nov 21, 2023 • edited Loading

marcorichetta commented May 13, 2024

Cross-compiling `linux/amd64` on M1 mac #165

Cross-compiling `linux/amd64` on M1 mac #165

lpgauth commented Feb 3, 2022 •

edited

Loading

abiosoft commented Feb 5, 2022 •

edited

Loading

nunovieira commented Apr 11, 2023 •

edited

Loading

divStar commented Sep 3, 2023 •

edited

Loading

ArvinB commented Nov 21, 2023 •

edited

Loading