@ChenhuiYuan01 is it only dotnet new that gets stuck or other commands as well? CC @MiYanni @joeloff if this is a templating issue.

@marcpopMSFT
Here are the commands we tried
1.dotnet--info , dotnet --version , dotnet --list-sdks -->Run successfully
2.dotnet new , dotnet restore, dotnet build , dotnet run , dotnet publish -->Stuck

This issue is also repro on 9.0.100-preview.4.24178.10 from https://github.com/dotnet/installer?tab=readme-ov-file.

@marcpopMSFT Here are the commands we tried 1.dotnet--info , dotnet --version , dotnet --list-sdks -->Run successfully 2.dotnet new , dotnet restore, dotnet build , dotnet run , dotnet publish -->Stuck

Seems like we're blocked on alpine arm but I don't know the how critical that platform is.

@nkolev92 @dsplaisted from the above list, it kind of looks like any command that hits nuget is hanging. dotnet new doesn't build but does do a restore after creating the project. Thoughts on this one?

I assume this is alpine arm64 only. If so, based on telemetry data I would not consider this a blocker. I checked the data and the arm64 alpine usage is <1% of all alpine usage and alpine usage is <1% of linux usage (from the data we have access to).

It could be something with restore, or with MSBuild. I would guess that it's something to do with the network access that restore does. We could do dotnet new and build with the options to skip restore to help validate where the hang is occurring. But really I think someone needs to attach a debugger or inspect a dump or something to figure out what's going on.

@marcpopMSFT @dsplaisted This is blocking Alpine Arm32 official .NET container images. Observing this with the same 9.0.100-preview.3.24175.24 version. We're not seeing this in Arm64 Docker images though. I put together this repro Dockerfile because I intended to file an issue, but found this existing issue instead:

FROM arm32v7/alpine:3.19

ENV \
    # Configure web servers to bind to port 8080 when present
    ASPNETCORE_HTTP_PORTS=8080 \
    # Enable detection of running in a container
    DOTNET_RUNNING_IN_CONTAINER=true \
    # Do not generate certificate
    DOTNET_GENERATE_ASPNET_CERTIFICATE=false \
    # Do not show first run text
    DOTNET_NOLOGO=true \
    # SDK version
    DOTNET_SDK_VERSION=9.0.100-preview.3.24175.24 \
    # Set the invariant mode since ICU package isn't included
    DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=true \
    # Enable correct mode for dotnet watch (only mode supported in a container)
    DOTNET_USE_POLLING_FILE_WATCHER=true \
    # Skip extraction of XML docs - generally not useful within an image/container - helps performance
    NUGET_XMLDOC_MODE=skip \
    # PowerShell telemetry for docker image usage
    POWERSHELL_DISTRIBUTION_CHANNEL=PSDocker-DotnetSDK-Alpine-3.19-arm32

RUN apk add --upgrade --no-cache \
        ca-certificates-bundle \
        \
        # .NET dependencies
        libgcc \
        libssl3 \
        libstdc++ \
        zlib \
        curl

# Install .NET SDK
RUN wget -O dotnet.tar.gz https://dotnetbuilds.azureedge.net/public/Sdk/$DOTNET_SDK_VERSION/dotnet-sdk-$DOTNET_SDK_VERSION-linux-musl-arm.tar.gz \
    && dotnet_sha512='c0a702b295f275b135d7fc845322b71ff9298fc35771fa0ae4118e5766d3d2c16a3658757a7b8cc41ec095a8c532b58322c91c1239ba851d48ba30480932fb95' \
    && echo "$dotnet_sha512  dotnet.tar.gz" | sha512sum -c - \
    && mkdir -p /usr/share/dotnet \
    && tar -oxzf dotnet.tar.gz -C /usr/share/dotnet \
    && rm dotnet.tar.gz \
    && ln -s /usr/share/dotnet/dotnet /usr/bin/dotnet

# Run any arbitrary .NET command
RUN dotnet --version \
    && dotnet help

It can be built with the command docker build --progress=plain --platform linux/arm/v7 -t repro -f ./Dockerfile .

I used an arm64 machine in DTL to repro it. It also hangs under Docker qemu emulation on an amd64 machine.

Related: dotnet/dotnet-docker#5309

@marcpopMSFT is this blocking .NET 9 Preview 3 release?

@lbussell if you hit this, did you happen to be able to collect a dump or attach a debugger to get a callstack? Could you try a dotnet build --no-restore and see if it doesn't hang then? I'm inclined to believe it's a restore hang but that would prove it so we can route to @dotnet/nuget-team .

And @NicoleWang001 no idea what the priority of this specific config is. That's a question for tactics.

@marcpopMSFT Sorry, my repro was not clear enough. The dotnet help line is what hangs. I can try to get a dump from the dotnet executable for that, if you think that will help.

You're saying it hangs when running dotnet --help? That line also has a dotnet new console on it so I assumed it was that. Please do if you can.

You're saying it hangs when running dotnet --help? That line also has a dotnet new console on it so I assumed it was that. Please do if you can.

@marcpopMSFT yes it hangs when running dotnet --help. Interestingly it hangs after printing the help command's text.

I cannot get a dump without some further help with this - dotnet tool install --global dotnet-dump hangs, obviously, and the dotnet dump tool does not provide a linux-musl-arm binary. Arm64 does not hang.

@lbussell can you install a different version of the SDK that's working in a different location in order to install the dump tool?

CC @elinor-fung @agocke in case the issue is in the host itself as --help shouldn't do much.

@lbussell can you install a different version of the SDK that's working in a different location in order to install the dump tool?

@marcpopMSFT

Here's what I'm trying -

1. Use latest known working SDK to install the dotnet-dump tool. Seems the last known good SDK was 9.0.0-preview.3.24162.31
2. Copy the working SDK and the dotnet-dump tool to the repro image.
3. Make sure the working version of .NET is on the PATH for the dotnet-dump tool to find.
4. Build repro image - docker build --progress=plain --platform linux/arm/v7 -t repro -f ./Dockerfile .
5. Run a dotnet command in the container to let it hang - docker run --rm -d --name repro-container repro /bin/sh -c '/usr/share/dotnet-broken/dotnet --help'
6. Confirm the dotnet --help process PID (it's usually 1) - docker exec repro-container /bin/sh -c 'export DOTNET_ROOT=/usr/share/dotnet-working/ && /root/.dotnet/tools/dotnet-dump ps'
7. Run dotnet-dump tool to collect a dump - docker exec repro-container /bin/sh -c 'export DOTNET_ROOT=/usr/share/dotnet-working/ && /root/.dotnet/tools/dotnet-dump collect --diag --type Mini -p 1'

The dotnet-dump tool hangs forever in this configuration, even trying to collect a "Mini" dump. /cc @mikem8361

Here's the updated Dockerfile I'm using to do this -

FROM mcr.microsoft.com/dotnet/nightly/sdk:9.0-preview-alpine3.19-arm32v7 as installer

RUN dotnet tool install --global dotnet-dump

FROM arm32v7/alpine:3.19

ENV ... same as above

RUN apk add ... same as above

COPY --from=installer [ "/usr/share/dotnet/", "/usr/share/dotnet-working/" ]
COPY --from=installer [ "/root/.dotnet/tools", "/root/.dotnet/tools" ]

# Install .NET SDK
RUN wget -O dotnet.tar.gz https://dotnetbuilds.azureedge.net/public/Sdk/$DOTNET_SDK_VERSION/dotnet-sdk-$DOTNET_SDK_VERSION-linux-musl-arm.tar.gz \
    && mkdir -p /usr/share/dotnet-broken \
    && tar -oxzf dotnet.tar.gz -C /usr/share/dotnet-broken \
    && rm dotnet.tar.gz \
    && ln -s /usr/share/dotnet-working/dotnet /usr/bin/dotnet

Per @agocke in tactics, moving to runtime repo.

@mangod9 can you take a look? not sure where this high cpu may be coming from. also including @elinor-fung

@ChenhuiYuan01, assuming this is a consistent repro are you able to capture a dump with dotnet-dump when the high cpu is occurring? Thanks

You're saying it hangs when running dotnet --help? That line also has a dotnet new console on it so I assumed it was that. Please do if you can.

@marcpopMSFT yes it hangs when running dotnet --help. Interestingly it hangs after printing the help command's text.

I cannot get a dump without some further help with this - dotnet tool install --global dotnet-dump hangs, obviously, and the dotnet dump tool does not provide a linux-musl-arm binary. Arm64 does not hang.

@mangod9 We could not get a dump either as the dotnet-dump hangs.
Also tried below, but still could not get a dump

1. On Alpine Arm32 container, install .NET 8.0->install tool dotnet-dump ->install SDK .NET 9.0.100-preview.3.24175.24 and run dotnet--help ->dotnet-dump hangs as well
2. On Alpine Arm32 container, install SDK .NET 9.0.100-preview.3.24175.24 and run dotnet--help. On host, install SDK .NET 9.0.100-preview.3.24175.24-> install tool dotnet-dump -> dotnet-dump failed with
   Unhandled exception: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
   ---> System.Net.Sockets.SocketException (111): Connection refused

@NicoleWang001 I would suggest to generate a core dump without any .NET tool help (using OS bundled native tools). Then run dotnet-dump on a similar environment (that is not that failing docker container, e.g. a real arm32v7 environment) to analyze that previously collected core dump.

I am experiencing hangs (stuck forever, manually aborted by me after 8 hours) for all dotnet commands when targeting arm32v7 once the SDK is installed (not if only the runtime is installed though), for .NET 6, 7, 8 and 9:

FROM mcr.microsoft.com/dotnet/nightly/sdk:9.0.100-preview.3-alpine3.19-arm32v7
WORKDIR /srv
RUN dotnet help

FROM mcr.microsoft.com/dotnet/sdk:8.0-alpine3.18-arm32v7
WORKDIR /srv
RUN dotnet help

FROM mcr.microsoft.com/dotnet/sdk:7.0-alpine3.18-arm32v7
WORKDIR /srv
RUN dotnet help

FROM mcr.microsoft.com/dotnet/sdk:6.0-alpine3.18-arm32v7
WORKDIR /srv
RUN dotnet help

docker build -f 'arm32v7-test.Dockerfile' --platform 'linux/arm32v7' --pull -t 'arm32v7-test:latest' ./empty/

Tested with a docker host that is Windows 10 x64 (with WSL2 integration to Ubuntu) or an Ubuntu 22.04/23.10 VM (VirtualBox, with a Windows 10 x64 host). So these are environments that use qemu to emulate arm32v7.

BTW, downloading and running the SDK in a custom build (buildroot) qemu busybox image targeting arm32v7 (so without docker) works fine. I have not yet compared my custom buildroot options against those of https://github.com/docker-library/busybox.

Here are results from COREHOST_TRACE=1:

• COREHOST_TRACE=1 output for dotnet help in docker before it hangs: dotnet_trace_01.txt
• COREHOST_TRACE=1 output for dotnet help in custom working busybox qemu build (without docker): dotnet_trace_02.txt

Differences are the HOST_RUNTIME_CONTRACT and the following lines that are output in the working build in addition to the output of the non-working docker build (the docker build hangs before those lines):

Launch host: /usr/share/dotnet/dotnet, app: /usr/share/dotnet/sdk/8.0.203/dotnet.dll, argc: 1, args: help,
--- Begin breadcrumb write
Directory core breadcrumbs [] was not specified or found
Fallback directory core breadcrumbs at [opt/corebreadcrumbs] was not found
Breadcrumb store was not obtained... skipping write.
Execute managed assembly exit code: 0x0

/cc @richlander

@lauxjpn the behavior for the 6.0, 7.0, and 8.0 Docker images is unexpected. Can you please post an issue for that in https://github.com/dotnet/dotnet-docker?

I have done some inital investigations. I have found the issue is related to the new exception handling that was enabled in preview 3 by default. There is some exception that occurs on a secondary thread during the shutdown of the process and the function that the new exception handling uses to iterate over stack frames keeps returning the same frame over and over. This is actually an explicit frame (InlinedCallFrame) which has its internal "next" pointer pointing to itself. I don't have any idea how that could happen at the moment, but that causes the infinite loop there.
Until the issue is fully understood and fixed, it can be mitigated by enabling the legacy exception handling by setting the env variable DOTNET_LegacyExceptionHandling=1.