New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Running dotnet core on Alpine for ARM32 #17468

Open
JensNordenbro opened this Issue Apr 7, 2018 · 40 comments

Comments

Projects
None yet
8 participants
@JensNordenbro
Copy link

JensNordenbro commented Apr 7, 2018

Due to its small footprint (OS image size), Alpine is suited for embedded scenarios on ARM32.

Things leading to this:

  • Today dotnet core is Community Supported on ARM32.

  • dotnet core 2.1 will support arm32 officially (Supported Operating Systems and Chip Architectures-section)

  • Today dotnet core 2.1 is running on Alpine for x86.

  • It is possible to run docker on an embedded system (for example ResinOS) and in order to do this efficiently a small container image is needed, hence the need for Alpine support.

  • It is expected to have symmetry in OS support between different CPU architectures where possible

(If/when this is implemented the docker images for running .net core application should be available. )

@RussKeldorph

This comment has been minimized.

Copy link
Member

RussKeldorph commented Apr 10, 2018

@janvorli

This comment has been minimized.

Copy link
Member

janvorli commented Apr 10, 2018

See my comment #9370 (comment)
We want to support ARM32 Alpine, but in order to do that, we need a way to cross build for Alpine arm on x64. The cross build mechanisms that we have in our build now works only for debian based distros and uses qemu-debootstrap. That would not work for Alpine.
In the comment above, I've identified a potential way to do that.

@janvorli janvorli added this to the Future milestone Apr 10, 2018

@ghost

This comment has been minimized.

Copy link

ghost commented Apr 19, 2018

Alpine arm32 architecture is called armhf in their docs, which supports armv6 and armv7 target architectures.

Unfortunately, they only provide armv6-alpine-linux-musleabi, which does not compile CoreCLR, as armv6 is not supported: #10605. I tried compiling coreclr by adding

elseif(CLR_CMAKE_PLATFORM_ALPINE_LINUX) 
  add_compile_options(-target armv6-alpine-linux-musleabihf)

in compileoptions.cmake, but compilation failed due to unsupported assembly instructions (dmb ish etc.)

I then searched and asked Alpine community over chat, apparently they considered/discussed it at some point but have "made a decision" to not provide the said armv7 official release; it is either armv6 for 32-bit or armv8 for 64-bit. This way they will not have to support two variants of armhf.

@ghost

This comment has been minimized.

Copy link

ghost commented Apr 19, 2018

One approach could be to build our own compiler toolchain and armv7 ABI using https://github.com/richfelker/musl-cross-make, then ship it with the ARM32 runtime package. But I think then all the external native dependencies would need to be compiled with same toolchain? Sounds complicated.

@janvorli

This comment has been minimized.

Copy link
Member

janvorli commented Apr 19, 2018

Oh, that's unfortunate. But as for your previous comment, why would we need to ship the toolchain? Wouldn't compiling the runtime bits with it be sufficient?

@ghost

This comment has been minimized.

Copy link

ghost commented Apr 19, 2018

I was thinking that we might need to ship the generated ABI libs (not toolchain). It makes sense that technically it should be sufficient to build and compile in a way that we shouldn't need to package any extra libs like that with our bins (it should be just work on alpine installation on armv7 device).

Out of the box, they support GCC. It would be ideal for us to generate armv7-alpine-linux-musleabihf, then we can use clang to compile with this abi target. Found related conversation: richfelker/musl-cross-make#34.

By the way, i used troyfontaine/armhf-alpinelinux (https://github.com/troyfontaine/armhf-alpinelinux) qemu alpine 3.7 docker to try compiling coreclr. Will test out musl-cross-make. Even with extra qemu-ish layers, it is super slim (6mbs)!

apk update
apk add cmake git bash build-base icu-dev lttng-ust-dev clang llvm-dev linux-headers
apk add lldb-dev --no-cache -X http://dl-cdn.alpinelinux.org/alpine/edge/testing
git clone https://github.com/dotnet/coreclr
cd coreclr

# apply musl ABI patch in compileoptions

./build.sh -cmakeargs -DCMAKE_CXX_COMPILER=$(which clang++)

(for some reason, cmake picks up /bin/c++ instead of /bin/clang++ in this docker)

@janvorli

This comment has been minimized.

Copy link
Member

janvorli commented Apr 20, 2018

(for some reason, cmake picks up /bin/c++ instead of /bin/clang++ in this docker)

I have seen cases when clang package has installed clang, but clang++-3.x or vice versa. In other words, the version was just on one of those and it was causing the same or similar issue. Adding a symbolic link to make it symmetric helped.

I was thinking that we might need to ship the generated ABI libs

I believe the ABI is the same and the only difference is in the supported instructions inside the binaries. So we should be able to build dotnet core using the self compiled toolchain, but the result should work on the Alpine as is.

@ghost

This comment has been minimized.

Copy link

ghost commented Apr 22, 2018

I made a simple test:

#include <stdio.h>
int main()
{
   asm volatile ("dmb ish" : : : "memory");
   return 0;
}

compiled with clang test.c -o test.exe -march=armv7-a on that docker and it worked (without -march=armv7-a, it gives the error). It seems like something is wrong with CMake configuration (something is overwriting CFlags).

My current patch is:

diff --git a/compileoptions.cmake b/compileoptions.cmake
index 9c352e8b..f87bfea4 100644
--- a/compileoptions.cmake
+++ b/compileoptions.cmake
@@ -66,6 +66,9 @@ if(CLR_CMAKE_PLATFORM_UNIX_ARM)
      add_definitions(-DARM_SOFTFP)
      add_compile_options(-mfloat-abi=softfp)
      add_compile_options(-target armv7-linux-gnueabi)
+   elseif(CLR_CMAKE_PLATFORM_ALPINE_LINUX)
+#     add_compile_options(-target armv7-alpine-linux-musleabihf)
+     set(CMAKE_C_FLAGS "-march=armv7-a ${CMAKE_C_FLAGS}")
    else()
      add_compile_options(-target armv7-linux-gnueabihf)
    endif(ARM_SOFTFP)
@ghost

This comment has been minimized.

Copy link

ghost commented Apr 23, 2018

Pushed one more commit to #17730, with that CoreCLR builds on ARM32 Alpine using steps described #17468 (comment).

@ghost ghost referenced this issue Apr 23, 2018

Merged

Add linux-musl build leg #17623

@ghost

This comment has been minimized.

Copy link

ghost commented Apr 25, 2018

I have experimented with chroot and finally made it work:

Using docker to get Ubuntu with nested virtualization enabled

docker run -it --privileged ubuntu:bionic

entered Ubuntu docker:

# update packages
apt update

# install some utils
apt install -y apt-utils wget < /dev/null

# download alpine chroot script (and verify its checksum)
wget https://raw.githubusercontent.com/alpinelinux/alpine-chroot-install/v0.8.0/alpine-chroot-install \
  && echo 'a3d7e2e3e63dfb8abcabb35829a6c8c18bdab082  alpine-chroot-install' | sha1sum -c

# make the script executable
chmod 777 alpine-chroot-install

# call script (see other options in that script, like to mount a directory, use `-i` option)
./alpine-chroot-install -a armhf -b v3.7 -m http://dl-cdn.alpinelinux.org/alpine

# enter chroot
/alpine/enter-chroot

entered alpine armv7 chroot:

# update packages
apk update

# install build prerequisites and utilities
apk add cmake git bash build-base icu-dev lttng-ust-dev clang llvm-dev linux-headers

# install lldb, which is in testing repository
apk add lldb-dev --no-cache -X http://dl-cdn.alpinelinux.org/alpine/edge/testing

# clone coreclr
git clone https://github.com/dotnet/coreclr
cd coreclr

# build
./build.sh -cmakeargs -DCMAKE_CXX_COMPILER=$(which clang++)

@janvorli, could you please help integrating it in CoreCLR's cross build scripts? The additional toolchain.cmake etc. are confusing, as there we are again setting ABI which is not strictly necessary (imo for other platforms as well)

@janvorli

This comment has been minimized.

Copy link
Member

janvorli commented Apr 25, 2018

@kasper3 that is awesome! Thank you so much for looking into it and making it work!
I'll be happy to do the integration.

@ghost

This comment has been minimized.

Copy link

ghost commented Apr 26, 2018

One upstream issue is fixed: alpinelinux/alpine-chroot-install#8, I have updated steps above to use v0.8.0 version of chroot script.

@sdmaclea

This comment has been minimized.

Copy link
Member

sdmaclea commented Apr 26, 2018

@janvorli Would you be willing to do this for arm64 also. I think alpine is interesting in arm64 docker scenarios because it is lightweight.

@janvorli

This comment has been minimized.

Copy link
Member

janvorli commented Apr 26, 2018

@sdmaclea sure, I think there would be only minor differences needed for arm64.

I'll get to it next week, I'm flying home from Redmond this afternoon.

@janvorli

This comment has been minimized.

Copy link
Member

janvorli commented May 9, 2018

Update: I have the build working for both arm and arm64, but I need to test it on a real device / container yet to make sure the compiled binaries are 100% correct.

@kasper3 the steps you have provided were really a great guideline. However, they were actually compiling under the qemu and so the compilation was really slow. So I have implemented the cross build the same way as we build for debian / ubuntu. That means that the build system's clang is used for building and the rootfs just provides the libraries / headers to compile / link against.
The only issue was that the LD sets the .interp section to point to the glibc loader instead of the musl one, since it knows nothing about musl. For the time being, I've added a linker option that asks it to set it explicitly to the musl one, but I am not sure if that's a correct solution.
I am sorry for the delay, but I was OOF this and the last week Monday and Tuesday.

If you are interested, you can check the current state here:
janvorli@0a03ff8

@ghost

This comment has been minimized.

Copy link

ghost commented May 9, 2018

Thanks @janvorli. It looks good to me, but if possible, I think it would be nice to request @rofl0r for the pull request review to get some holistic feedback. 😎

A bit off-topic, should we try to merge Android rootfs script into the same one to easily keep the implementation for all arm platforms in sync?

@ghost

This comment has been minimized.

Copy link

ghost commented Jun 6, 2018

CoreCLR part is done in #18234. Should this be closed from CoreCLR and open in CoreFX / coreSetup for remaining work?

@janvorli

This comment has been minimized.

Copy link
Member

janvorli commented Jun 7, 2018

CoreFX PR for the same thing is pending: dotnet/corefx#30182. I'll make the same change in core-setup soon.

@janvorli

This comment has been minimized.

Copy link
Member

janvorli commented Jun 7, 2018

And here core-setup: dotnet/core-setup#4203

@JensNordenbro

This comment has been minimized.

Copy link
Author

JensNordenbro commented Jun 7, 2018

Will this work be for 2.1 lts or 3.0?

@sdmaclea

This comment has been minimized.

Copy link
Member

sdmaclea commented Jun 7, 2018

Definitely not for 2.1. 2.2 or 3.0

@janvorli

This comment has been minimized.

Copy link
Member

janvorli commented Jun 7, 2018

@sdmaclea why not for 3.0? We have not discussed it yet, but 3.0 seems like a very reasonable target.

@ghost

This comment has been minimized.

Copy link

ghost commented Jun 7, 2018

Thanks @janvorli!
I think the remaining work is to get CLI story for ARM32 set: dotnet/cli#8998? It seems like @echesakovMSFT hasn't committed to it yet, do you know what are the predicaments in crossgening arm32 on x64?

@sdmaclea

This comment has been minimized.

Copy link
Member

sdmaclea commented Jun 7, 2018

Definitely not for 2.1. 2.2 or 3.0

@sdmaclea why not for 3.0?

The period after the 2.1 was too subtle. Should have said "Definitely not for 2.1. Either for 2.2 or 3.0"

@janvorli

This comment has been minimized.

Copy link
Member

janvorli commented Jun 7, 2018

Ah, makes sense.

@echesakovMSFT

This comment has been minimized.

Copy link
Member

echesakovMSFT commented Jun 7, 2018

@kasper3 Answering your question about crossgening arm32 on x64 - I was distracted during 2.1 shipping. Now it has been shipped and I am back actively working on this problem.

@janvorli

This comment has been minimized.

Copy link
Member

janvorli commented Jun 7, 2018

@kasper3 actually, there is more to do. First we need to enable official builds in coreclr, corefx and core-setup repos so that nuget packages are built and published.

@JensNordenbro

This comment has been minimized.

Copy link
Author

JensNordenbro commented Aug 17, 2018

Any progress on this?

@mshgh

This comment has been minimized.

Copy link

mshgh commented Sep 21, 2018

@janvorli just curious has the target been discussed? Any chance for this to make it into 2.2? ;) Or at least the 3.0 mentioned above as "likely very reasonable target"?

@richlander

This comment has been minimized.

Copy link
Member

richlander commented Nov 17, 2018

It is unclear to me what is left to discuss here ...

Both the .NET Core 2.1 runtime and SDK are now working on Linux+ARM32, including Alpine. Last time I checked, there were still bugs on Linux+ARM64 for .NET Core 3.0, but Linux+ARM64 is targeted for that release.

@mshgh

This comment has been minimized.

Copy link

mshgh commented Nov 17, 2018

@richlander I have checked the 2.1 download page and it seems to me the version requested in this issue is still not available. But, honestly, I'd be more then happy if you prove me wrong.

When looking into the Linux section it is not immediately obvious to me which package should be downloaded when. Let me explain. There are two sections Linux glibc and Linux musl this is great, but this doesn't tell you which architecture these packages are for. Then if you check the package names this is 2.1.0-linux-x64 and 2.1.0-musl-x64 respectively. This reveals architecture - Intel 64bits. Note the first package would be better named 2.1.0-glibc-x64. And there is no 32bit version, which is not big deal.

Now there is another section called Linux ARM with 32/64bit packages named 2.1.0-linux-arm and 2.1.0-linux-arm64. If I follow the naming convention chosen for Intel this would be glibc based => 2.1.0-glibc-armXX, but there is not musl versions for these packages. And Alpine is based on musl. So package I am looking for is 2.1.0-musl-arm, respectively to use the name for architecture as Alpine packages do it the name would be 2.1.0-musl-armhf.

In the end the names are not critical, if the binaries deliver what is necessary. And the milion dollar question is. Does such package exist?

P.S. Yes, on 32bit ARM Alpine can be installed glibc support in parallel to musl and then the 2.1.0-linux-arm package should work, but this is not what I am looking for. and I am sure neither author of this issue.

@JensNordenbro

This comment has been minimized.

Copy link
Author

JensNordenbro commented Nov 17, 2018

I have the same experience and also there seems to be no docker Images available.

@janvorli

This comment has been minimized.

Copy link
Member

janvorli commented Nov 19, 2018

@richlander we still don't have official builds for Alpine ARM / ARM64. So we have neither SDK nor CLI yet. In the past, I have just enabled the crossbuild support for Alpine ARM / ARM64 in our repos, but I haven't added the official builds.

@mshgh

This comment has been minimized.

Copy link

mshgh commented Nov 19, 2018

@janvorli no pressure ;) but the follow up question is obvious. Are there any plans to add the official builds for Alpine ARM?

@hjc4869

This comment has been minimized.

Copy link

hjc4869 commented Nov 19, 2018

Vote +1 for official build. I test my code using linux-arm64 NuGet packages everyday even though it's not yet stable and officially supported. It's much more convenient to have official linux-musl-arm(64) packages published.

@JensNordenbro

This comment has been minimized.

Copy link
Author

JensNordenbro commented Nov 19, 2018

Vote +1; @janvorli ; Having the runtime is good for many scenarios; sdk + cli + offical docker images can follow later.

@hjc4869

This comment has been minimized.

Copy link

hjc4869 commented Dec 27, 2018

It seems that Alpine ARM64 will be officially supported in .NET Core 3.0. Unfortunately Alpine ARM32 is not on the list.

@mshgh

This comment has been minimized.

Copy link

mshgh commented Dec 27, 2018

Good catch @hjc4869, unfortunately I cannot see any link to "ARM64 Alpine" in the .NET 3.0 downloads page. Only "x64 Apline" one is there. Or did you have better luck locating the binaries?

@hjc4869

This comment has been minimized.

Copy link

hjc4869 commented Dec 28, 2018

Binary and docker images are not released yet, I guess they'll come later.

@janvorli

This comment has been minimized.

Copy link
Member

janvorli commented Jan 2, 2019

@hjc4869 I am planning to enable Alpine for both arm32 and arm64. @mshgh we have to enable building Alpine ARM/ARM64 nuget packages yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment