Running any dotnet CLI command (e.g. dotnet help) hangs on official .NET SDK 6/7/8/9 arm32v7 musl docker containers (and qemu-arm in general)

[lauxjpn]

I am experiencing hangs (stuck forever, manually aborted by me after 8 hours) for all dotnet commands when targeting arm32v7 once the SDK is installed (not if only the runtime is installed though), for .NET 6, 7, 8 and 9:

FROM mcr.microsoft.com/dotnet/nightly/sdk:9.0.100-preview.3-alpine3.19-arm32v7
WORKDIR /srv
RUN dotnet help

FROM mcr.microsoft.com/dotnet/sdk:8.0-alpine3.18-arm32v7
WORKDIR /srv
RUN dotnet help

FROM mcr.microsoft.com/dotnet/sdk:7.0-alpine3.18-arm32v7
WORKDIR /srv
RUN dotnet help

FROM mcr.microsoft.com/dotnet/sdk:6.0-alpine3.18-arm32v7
WORKDIR /srv
RUN dotnet help

docker build -f 'arm32v7-test.Dockerfile' --platform 'linux/arm32v7' --pull -t 'arm32v7-test:latest' ./empty/

Tested with a docker host that is Windows 10 x64 (with WSL2 integration to Ubuntu) or an Ubuntu 22.04/23.10 VM (VirtualBox, with a Windows 10 x64 host). So these are environments that use qemu to emulate arm32v7.

BTW, downloading and running the SDK in a custom build (buildroot) qemu busybox image targeting arm32v7 (so without docker) works fine. I have not yet compared my custom buildroot options against those of https://github.com/docker-library/busybox.

---

Here are results from COREHOST_TRACE=1:

• COREHOST_TRACE=1 output for dotnet help in docker before it hangs: dotnet_trace_01.txt
• COREHOST_TRACE=1 output for dotnet help in custom working busybox qemu build (without docker): dotnet_trace_02.txt

Differences are the HOST_RUNTIME_CONTRACT and the following lines that are output in the working build in addition to the output of the non-working docker build (the docker build hangs before those lines):

Launch host: /usr/share/dotnet/dotnet, app: /usr/share/dotnet/sdk/8.0.203/dotnet.dll, argc: 1, args: help,
--- Begin breadcrumb write
Directory core breadcrumbs [] was not specified or found
Fallback directory core breadcrumbs at [opt/corebreadcrumbs] was not found
Breadcrumb store was not obtained... skipping write.
Execute managed assembly exit code: 0x0

/cc @richlander

Originally posted by @lauxjpn in dotnet/runtime#100536 (comment)

---

I now also tried to run a HelloWord console app from a vanilla arm32v7/alpine:3.18, with a manually build dotnet runtime and the necessary package dependencies, and are getting the same issue. So this is not coupled to only the .NET SDK docker images, but is a general issue with the dotnet executable on arm32v7 docker images (or at least alpine/musl based ones).

Also, AOT compiled apps work just fine.

---

I now also tried this setup without docker, just with qemu-arm directly on an Ubuntu system, which also fails (while qemu-aarch64 works). So this is not docker specific but qemu-arm specific (or qemu specific). As in the docker scenario, it is only the musl based dotnet runtime that hangs (the non-musl based one works fine).

/cc @lbussell

Output of docker version

Client:
 Cloud integration: v1.0.35+desktop.11
 Version:           25.0.3
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        4debf41
 Built:             Tue Feb  6 21:13:02 2024
 OS/Arch:           windows/amd64
 Context:           default

Server: Docker Desktop 4.28.0 (139021)
 Engine:
  Version:          25.0.3
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       f417435
  Built:            Tue Feb  6 21:14:25 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker info

Client:
 Version:    25.0.3
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.1-desktop.4
    Path:     C:\Program Files\Docker\cli-plugins\docker-buildx.exe
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.6-desktop.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-compose.exe
  debug: Get a shell into any image or container. (Docker Inc.)
    Version:  0.0.24
    Path:     C:\Program Files\Docker\cli-plugins\docker-debug.exe
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-dev.exe
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.22
    Path:     C:\Program Files\Docker\cli-plugins\docker-extension.exe
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.4
    Path:     C:\Program Files\Docker\cli-plugins\docker-feedback.exe
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.0.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-init.exe
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-sbom.exe
  scout: Docker Scout (Docker Inc.)
    Version:  v1.5.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-scout.exe

Server:
 Containers: 41
  Running: 2
  Paused: 0
  Stopped: 39
 Images: 57
 Server Version: 25.0.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
 Kernel Version: 5.15.146.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 23.47GiB
 Name: docker-desktop
 ID: fb44f0c3-8984-458b-b79f-5007193d7663
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
WARNING: daemon is not using the default seccomp profile

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[lbussell]

[Triage] You are running .NET through QEMU emulation, which is not supported for any current version of .NET, so the behavior you're seeing is undefined - please see the official support documentation:

• https://github.com/dotnet/core/blob/main/release-notes/6.0/supported-os.md#qemu
• https://github.com/dotnet/core/blob/main/release-notes/7.0/supported-os.md#qemu
• https://github.com/dotnet/core/blob/main/release-notes/8.0/supported-os.md#qemu

[lauxjpn]

@lbussell Is there any info somewhere about the issues that make QEMU a non-supported scenario? As I mentioned in the OP, it seems that only arm32v7 with musl doesn't work, while other combinations actually do work.

[lbussell]

@lauxjpn You can read more about it on the qemu repo: https://gitlab.com/qemu-project/qemu/-/issues/249. Also a search in the runtime repo for qemu will surely bring up lots of results.

If you need to build arm32 or arm64 .NET containers using an amd64 machine, you can consult this blog post, this discussion post or the samples directory in this repo. Some of the content may be geared towards building amd64 containers using Apple Silicon, but the cross-compilation functionality goes both ways.

If you need to run arm32/arm64 containers on your amd64 machine, we'd be interested in hearing more about your scenario.

[lauxjpn]

You can read more about it on the qemu repo: https://gitlab.com/qemu-project/qemu/-/issues/249.

@lbussell Thanks. During my research, I already came across https://gitlab.com/qemu-project/qemu/-/issues/249, but it seems to describe a QEMU bug with the implementation of the iretq instruction for x64 guests, not arm32v7 guests. So currently, I don't yet see that this issue is relevant to the arm32v7 hang in the .NET runtime.

If you need to build arm32 or arm64 .NET containers using an amd64 machine [...]

Cross-compiling for arm32v7 on x64/amd64 works fine. It's really just the runtime that is the issue.

If you need to run arm32/arm64 containers on your amd64 machine, we'd be interested in hearing more about your scenario.

I just want to run dotnet apps in an emulated arm32v7 environment, so that I don't have to use actual am32v7 hardware for testing all the time.

[lbussell]

I will defer to @jkotas for the actual details of why the runtime doesn't work on QEMU.

[jkotas]

Our experience has been that QEMU is too unstable to run .NET runtime. We tried to use QEMU to run .NET runtime tests at one point and found that QEMU is unsuitable. We (.NET team) do not have expertise and capacity to chase down all bugs in QEMU to make .NET runtime work well on it.

BTW: We do support the Rosetta emulator on Apple and the x64 emulator that comes with Windows Arm64. These emulators are very stable, and they have official support channels, so we do not have a problem with declaring .NET runtime as supported on those.

[lauxjpn]

@jkotas Thanks for clarifying. I will take a look at what I can do about that.