Cross-compiling for linux/arm64 causes segmentation fault on hosts with Linux kernels newer than 6.8.0-49-generic

[JVALPASS]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem

Description

Cross-compiling linux/arm64 Docker images using docker buildx results in segmentation faults during the build process on host systems running Linux kernel versions newer than 6.8.0-49-generic. Builds are successful on kernel 6.8.0-49-generic.

Expected behaviour

The docker buildx build process for linux/arm64 should complete successfully without segmentation faults on hosts with recent Linux kernels, including those newer than 6.8.0-49-generic.

Actual behaviour

The build process is interrupted by a segmentation fault when the host kernel is newer than 6.8.0-49-generic. The build completes successfully on kernel 6.8.0-49-generic.

Buildx version

github.com/docker/buildx v0.20.0 8e30c46

Docker info

Output from `docker version` (as `docker info` output was not directly provided):
Client: Docker Engine - Community
 Version:         27.5.1
 API version:     1.47
 Go version:      go1.22.11
 Git commit:      9f9e405
 Built:           Wed Jan 22 13:41:48 2025
 OS/Arch:         linux/amd64
 Context:         default

Server: Docker Engine - Community
 Engine:
  Version:         27.5.1
  API version:     1.47 (minimum version 1.24)
  Go version:      go1.22.11
  Git commit:      4c9b3b0
  Built:           Wed Jan 22 13:41:48 2025
  OS/Arch:         linux/amd64
  Experimental:    false
 containerd:
  Version:         1.7.25
  GitCommit:       bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
 runc:
  Version:         1.2.4
  GitCommit:       v1.2.4-0-g6c52b3f
 docker-init:
  Version:         0.19.0
  GitCommit:       de40ad0

Builders list

vpassamano@iitavr002lw013u:~/Desktop/ros1-ros2-zed-jetson-docker/ros1_noetic$ docker buildx ls
NAME/NODE        DRIVER/ENDPOINT                   STATUS    BUILDKIT   PLATFORMS
mybuilder*       docker-container                                       
 \_ mybuilder0    \_ unix:///var/run/docker.sock   running   v0.20.2    linux/amd64 (+4), linux/arm64, linux/arm (+2), linux/ppc64le, (4 more)
default          docker                                                 
 \_ default       \_ default                       running   v0.18.2    linux/amd64 (+4), linux/arm64, linux/arm (+2), linux/ppc64le, (7 more)

Configuration

Invoked Commands (general setup and build):

1. QEMU setup (run once on host): sudo docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
2. Buildx builder creation (if not existing): docker buildx create --bootstrap --name mybuilder --driver docker-container --use
3. Build command (executed on a host with a problematic kernel newer than 6.8.0-49-generic):

   docker buildx build --builder mybuilder --platform linux/arm64 --tag my-arm64-image --load .

Dockerfile/Project:
A minimal Dockerfile to reliably reproduce the segmentation fault on problematic kernels is not yet available. The issue is observed when building complex ROS 2 Humble images for NVIDIA Jetson Orin (target: linux/arm64), which involve compiling numerous C++ packages (e.g., ZED SDK related components, PCL, OpenCV) and their dependencies. The segmentation fault occurs during RUN steps that execute compilation or other computationally intensive processes within the QEMU-emulated linux/arm64 environment.

Build logs

274.6 c++: internal compiler error: Segmentation fault signal terminated program cc1plus
...
274.6 make[2]: *** [rospack/CMakeFiles/rospackexe.dir/build.make:76: rospack/CMakeFiles/rospackexe.dir/src/rospack_main.cpp.o] Error 4

Additional info

Host System Details (where kernel 6.8.0-49-generic works, and newer ones fail):

• Host OS: Ubuntu 24.04.1 LTS (Codename: noble)
• Host Kernel (Last Known Good): Linux iitavr002lw013u 6.8.0-49-generic #49-Ubuntu SMP PREEMPT_DYNAMIC Mon Nov 4 02:06:24 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux (Builds complete successfully with this kernel).
• Host Kernel (Problematic): kernel_version[>=6.8.0-51-generic]
• QEMU Package Version: qemu-user-static, Version: 1:8.2.2+ds-0ubuntu1.6
• QEMU Binary Version (aarch64): qemu-aarch64 version 8.2.2 (Debian 1:8.2.2+ds-0ubuntu1.6)

Host Cross-Compilation Toolchain (present on the system, from the setup with the working kernel):

• aarch64-linux-gnu-gcc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0
• aarch64-linux-gnu-g++ (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0
• libc6-dev-arm64-cross: 2.39-0ubuntu8cross1
• libstdc++-13-dev-arm64-cross: 13.3.0-6ubuntu2~24.04cross1

Hypothesis:
The issue might stem from an incompatibility or interaction between newer Linux host kernels (those tested being > 6.8.0-49-generic), the binfmt_misc mechanism, and QEMU's user-mode emulation for aarch64. docker buildx relies on this underlying stack for cross-platform builds. Since Buildx is the primary user-facing tool orchestrating this workflow, this issue is being reported here for triage and potential investigation into the interaction.

Any insights, potential workarounds (other than downgrading the host kernel, if possible), or guidance on further debugging this segmentation fault during cross-compilation would be greatly appreciated.

[Adrian-Stanea]

Hi, any update on this?

Currently I'm facing the same issue where I'm trying to emulate linux/arm64 in a Docker build that also happens to compile ROS2 packages from source. I'm running Linux Kernel 6.8.0-60-generic.

The issue seems to apply when running on non-native systems since building the same Dockerfile works on an ARM64 system but fails on x86 with emulation.

Also I get this error: qemu: uncaught target signal 11 (Segmentation fault) - core dumped which appears in a non-deterministic manner when installing packages from apt. The issue also is present in Github Actions runners.

[steven-swanbeck]

I am facing this same issue when compiling C++ code for linux/arm64 on an amd64 machine with Linux Kernel version 6.8.0-60-generic.

My experience is similar to above, but I want to emphasize the non-deterministic nature of the errors.

I only ever observe the issue when compiling source code and it appears to be stochastic, failing at different stages and usually with this error: c++: internal compiler error: Segmentation fault signal terminated program cc1plus.

When building extremely small libraries from source, I sometimes get lucky and make it past build stages without a seg fault, so all I can tell is the likelihood of a seg fault seems to increase with compilation time/size of the project being built.