Long time build with remote node

[asamofal]

Hi!

I'm trying to set up a multi-arch building but the remote node (for amd64) doesn't speed up my build.

MacBook Pro M1 Max (32GB memory). Docker-desktop 4.16.1 (buildx 0.10.0).
Dockerfile:

FROM composer:2.4.1 AS composer
FROM mlocati/php-extension-installer:2.0.1 AS php-ext-installer
FROM php:8.1.9-fpm

RUN apt-get update && apt-get install --no-install-recommends -y \
    sudo \
    host telnet iputils-ping \
    nano vim \
    git \
    && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*;

COPY --from=php-ext-installer /usr/bin/install-php-extensions /usr/local/bin/
RUN install-php-extensions \
    mysqli \
    pdo_mysql \
    opcache \
    gd \
    bcmath \
    exif \
    zip \
    pcntl \
    imagick-3.7.0 \
    xdebug-3.1.5;

...

There are some other instructions in Dockerfile but it doesn't matter. There's only one really time-consuming action: install-php-extensions - script that installs (compiles) PHP extensions. That step takes most of the building time.

My builder instance:

docker buildx ls
NAME/NODE              DRIVER/ENDPOINT           STATUS  BUILDKIT PLATFORMS
local-remote-builder * docker-container
  local                desktop-linux             running v0.11.0  linux/arm64, linux/amd64, linux/amd64/v2, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6
  server               ssh://user@server         running v0.11.0  linux/amd64*, linux/amd64/v2, linux/amd64/v3, linux/amd64/v4, linux/386

Building:
docker buildx build --push --platform linux/amd64,linux/arm64 -t local-registry-name:php:8.1.9 .

I see that arm64 version finishes in ~200s, it's ok. But then I'm waiting for amd64 build ~1200s, in the logs I see that all of this time it's compiling php extensions.

If I run docker buildx build --platform linux/amd64 --no-cache . on the server itself - the build takes ~202s.
To test if I really use a remote node during the building - I broke the ssh connection, the build stuck at the start, and threw me an error, so it means the remote node is really in use.

So I can't get the point of why building on the remote node is so slow (in fact, the same time I'm getting when building amd64 image with qemu locally on my MacBook).

[asamofal]

Ok, I've noticed something...

My local node supports amd64 builds, so maybe I really building amd64 image locally (I guess builder pickups the first node that supports the requested platform).
I create builder instance like this:

Create builder and node: 
docker buildx create --name local-remote-builder --node local --platform linux/arm64

Add remote node:
docker buildx create --name local-remote-builder --append --node server --platform linux/amd64 ssh://user@server

Activate and bootstrap the builder
docker buildx use local-remote-builder
docker buildx inspect --bootstrap

My builder instance before bootstrap:

NAME/NODE              DRIVER/ENDPOINT           STATUS   BUILDKIT PLATFORMS
local-remote-builder * docker-container
  local                desktop-linux             inactive          linux/arm64*
  server               ssh://user@server         inactive          linux/amd64*

But bootstrapping adds additional platforms to my local node:

Name:          local-remote-builder
Driver:        docker-container
Last Activity: 2023-01-24 09:10:43 +0000 UTC

Nodes:
Name:      local
Endpoint:  desktop-linux
Status:    running
Buildkit:  v0.11.0
Platforms: linux/arm64*, linux/amd64, linux/amd64/v2, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6

Name:      server
Endpoint:  ssh://user@server
Status:    running
Buildkit:  v0.11.0
Platforms: linux/amd64*, linux/amd64/v2, linux/amd64/v3, linux/amd64/v4, linux/386

Why is it so? How to keep local node to be responsible only for arm64 builds?

UPD: oh, linux/amd64* means this node is prioritized for the platform. Then I don't have any ideas again.

[asamofal]

There's no load on the server during the build process, and docker buildx du shows:

Reclaimable:	0B
Total:		0B

So I think my remote node was just not picked up by the builder. Why?

[pierm8]

Same setup as you and same problem, buildx ignores platform preferences and assigns builds to the first node that supports the platform.

It seems to be a problem with buildx version v0.10.0. I fixed it by rolling back to Docker Desktop 4.15.0 with buildx v0.9.1. Everything works fine without reconfiguring the builder and now linux/amd64 builds are done on x86 server.

❯ docker --version
Docker version 20.10.21, build baeda1f

❯ docker buildx version
github.com/docker/buildx v0.9.1 ed00243a0ce2a0aee75311b06e32d33b44729689

❯ docker buildx inspect
Name:   localremote
Driver: docker-container

Nodes:
Name:      macmini
Endpoint:  unix:///var/run/docker.sock
Status:    running
Buildkit:  v0.11.1
Platforms: linux/arm64*, linux/riscv64*, linux/ppc64le*, linux/s390x*, linux/mips64le*, linux/mips64*, linux/arm/v7*, linux/arm/v6*, linux/amd64, linux/amd64/v2, linux/386

Name:      server
Endpoint:  ssh://server
Status:    running
Buildkit:  v0.11.1
Platforms: linux/amd64*, linux/amd64/v2*, linux/amd64/v3*, linux/386*