Archery docker: multiplatform support on arm64

[asfimport]

It seems right now is not possible to build/run arrow docker containers with arch=arm64 (the default platform is amd64)

I tried first with this command:

 

ARCH=arm64 archery docker run conda-cpp-valgrind

 

and got this error:

 

[+] Running 0/1
 ⠿ conda Error                                                                                                                                                                                                                                                                      1.8s
Pulling conda: Error response from daemon: manifest for apache/arrow-dev:arm64-conda not found: manifest unknown: manifest unknown
[+] Running 0/1
 ⠿ conda-cpp Error                                                                                                                                                                                                                                                                  1.7s
Pulling conda-cpp: Error response from daemon: manifest for apache/arrow-dev:arm64-conda-cpp not found: manifest unknown: manifest unknown
[+] Building 1.1s (3/3) FINISHED
 => [internal] load build definition from conda.dockerfile                                                                                                                                                                                                                          0.0s
 => => transferring dockerfile: 38B                                                                                                                                                                                                                                                 0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                   0.0s
 => => transferring context: 35B                                                                                                                                                                                                                                                    0.0s
 => ERROR [internal] load metadata for docker.io/arm64/ubuntu:18.04                                                                                                                                                                                                                 1.0s
------
 > [internal] load metadata for docker.io/arm64/ubuntu:18.04:
------
failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to create LLB definition: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Error: `docker-compose --file /arrow/docker-compose.yml build --build-arg BUILDKIT_INLINE_CACHE=1 conda` exited with a non-zero exit code 17, see the process log above.
The docker-compose command was invoked with the following parameters:
Defaults defined in .env:
  ALMALINUX: 8
  ALPINE_LINUX: 3.16
  ARCH: amd64
  ARCH_ALIAS: x86_64
  ARCH_SHORT: amd64
  ARROW_R_DEV: TRUE
  BUILDKIT_INLINE_CACHE: 1
  CLANG_TOOLS: 12
  COMPOSE_DOCKER_CLI_BUILD: 1
  CONAN: gcc10
  CUDA: 11.0.3
  DASK: latest
  DEBIAN: 11
  DEVTOOLSET_VERSION: -1
  DOCKER_BUILDKIT: 1
  DOCKER_VOLUME_PREFIX:
  DOTNET: 6.0
  FEDORA: 35
  GCC_VERSION:
  GO: 1.17
  HDFS: 3.2.1
  JDK: 8
  KARTOTHEK: latest
  LLVM: 13
  MAVEN: 3.5.4
  NODE: 16
  NUMBA: latest
  NUMPY: latest
  PANDAS: latest
  PYTHON: 3.8
  PYTHON_WHEEL_WINDOWS_IMAGE_REVISION: 2022-06-12
  R: 4.2
  REPO: apache/arrow-dev
  R_CUSTOM_CCACHE: false
  R_IMAGE: ubuntu-gcc-release
  R_ORG: rhub
  R_PRUNE_DEPS: FALSE
  R_TAG: latest
  SPARK: master
  STATICCHECK: v0.2.2
  TURBODBC: latest
  TZ: UTC
  UBUNTU: 20.04
  ULIMIT_CORE: -1
  VCPKG: 38bb87c
Archery was called with:
  export ARCH=arm64

 

Then, I tried to use the docker buildx plugin with this command:

 

ARCH=arm64 archery docker run --using-docker-buildx conda-cpp-valgrind

 

and got this error:

 

Error response from daemon: manifest for apache/arrow-dev:arm64-conda not found: manifest unknown: manifest unknown
docker pull apache/arrow-dev:arm64-conda exited with non-zero exit code 1
Error response from daemon: manifest for apache/arrow-dev:arm64-conda-cpp not found: manifest unknown: manifest unknown
docker pull apache/arrow-dev:arm64-conda-cpp exited with non-zero exit code 1
[+] Building 0.0s (0/0)
ERROR: cache export feature is currently not supported for docker driver. Please switch to a different driver (eg. "docker buildx create --use")
Error: docker buildx build --build-arg BUILDKIT_INLINE_CACHE=1 --build-arg arch=arm64 --cache-from type=registry,ref=apache/arrow-dev:arm64-conda-cache --cache-to type=registry,ref=apache/arrow-dev:arm64-conda-cache,mode=max --output type=docker -f /arrow/ci/docker/conda.dockerfile -t apache/arrow-dev:arm64-conda /arrow exited with non-zero exit code 1

My system env is:

 

uname -a
Darwin MacBook-Pro.local 21.6.0 Darwin Kernel Version 21.6.0: Mon Aug 22 20:19:52 PDT 2022; root:xnu-8020.140.49~2/RELEASE_ARM64_T6000 arm64
 
docker --version
Docker version 20.10.17, build 100c701

docker compose version
Docker Compose version v2.10.2

 

 

Reporter: Percy Camilo Triveño Aucahuasi / @aucahuasi

Original Issue Attachments:

• with-buildx.txt.log.sh
• without-buildx.txt.log.sh

_{Note: This issue was originally created as ARROW-17765. Please see the migration documentation for further details.}

[asfimport]

Kouhei Sutou / @kou:
Could you try ARCH=arm64v8 instead?

[asfimport]

Percy Camilo Triveño Aucahuasi / @aucahuasi:
Thanks for the advice, unfortunately it seems didn't work.
I ran:

ARCH=arm64v8 archery docker run conda-cpp-valgrind bash

with this error: without-buildx.txt.log.sh.
And also, I ran using the buildx plugin with no cache

ARCH=arm64v8 archery docker run --using-docker-buildx --no-cache conda-cpp-valgrind bash

and got this error: with-buildx.txt.log.sh