New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enabling Linux ARM32 for .NET Core #725

Closed
hqueue opened this Issue Dec 5, 2016 · 155 comments

Comments

Projects
None yet
@hqueue
Member

hqueue commented Dec 5, 2016

Host: Ubuntu 14.04 x64
Target: Ubuntu 14.04 or(and) 16.04 ARM, Ubuntu 14.04 ARM Softfp and Tizen

To enable core-setup for Linux ARM32, we are going to follow steps suggested in PR #712.

ARM hardfp

Common

  • stable default clang version for cross compilation (#1411)
  • Use same logic to prepare rootfs across dotnet (#1432)

CoreCLR

CoreFX

core-setup
For core-setup, we will follow steps below.

  • Add support for Linux Arm32 RID (PR #712)
  • Preparing ROOTFS for cross-compilation Infrastructure (issue #729, PR #747)
  • Enable cross compiling corehost for Linux/ARM (issue #723, PR #749, #766, #812 )
  • Update build script to enable cross building (PR #712)
  • Update build script to enable downloading and packaging for Linux/ARM (PR #712)
  • Build nuget packages for the host and Microsoft.NETCore.App metapackage (PR #712)
  • Make clang3.6 default toolset for cross arm32 build (issue #1322)
  • Enable CI support for Linux Arm32 (issue #790 #1512)
  • Enable Pipeline build support for Linux Arm32 (issue #789)
  • Update Core-Setup readme.md to reflect the download location of Linux Arm32 archive binaries.

For example, we want to build core-setup for Linux/ARM using following command.

$ ./build.sh  --env-vars DISABLE_CROSSGEN=1,TARGETPLATFORM=arm,TARGETRID=ubuntu.14.04-arm,CROSS=1,ROOTFS_DIR=/home/jyoung/git/dotnet/rootfs-coreclr/arm/

ASP.NET

ARM softfp

We will use armel for arm-softfp through out the dotnet.

Common

  • stable default clang version for cross compilation (#1411)
  • Use same logic to prepare rootfs across dotnet (#1432)

Debian.8

CoreFX - Part1

CoreCLR

CoreFX - Part2

  • Use armel for arm-softfp through out the dotnet. (dotnet/corefx#14803)
  • Prepare ROOTFS for cross-compilation Infrastructure (already available)

core-setup
For core-setup, we will follow steps below.

  • Use armel for arm-softfp through out the dotnet (#1025)
  • Generate nupkgs and tarball for debian.8-armel (#1025)
./build.sh --skiptests --env-vars DISABLE_CROSSGEN=1,TARGETPLATFORM=armel,TARGETRID=debian.8-armel,CROSS=1,ROOTFS_DIR=/home/jyoung/git/dotnet/rootfs/armel

Tizen 4.0.0

CoreFX - Part1

  • Add RID for Tizen

CoreCLR

CoreFX - Part2

core-setup
For core-setup, we will follow steps below.

  • Prepare ROOTFS for cross-compilation Infrastructure (#1025)
  • Generate nupkgs and tarball for tizen.4.0.0-armel (#1025)
  • Enable Pipeline build support for Linux armel (Tizen) (#1566)

dotnet-ci

./build.sh -c Release --skiptests --env-vars DISABLE_CROSSGEN=1,TARGETPLATFORM=armel,TARGETRID=tizen.4.0.0-armel,CROSS=1,ROOTFS_DIR=/home/jyoung/git/dotnet/rootfs/armel-tizen

Results

ubuntu.14.04-arm (Latest build is available at https://github.com/dotnet/core-setup#daily-builds)
dotnet-ubuntu.14.04-arm.1.2.0-beta-001291-00.tar.gz (Last updated on Jan 19)
dotnet-sdk-ubuntu.14.04-arm.1.0.0-preview5-004431.tar.gz
ubuntu.16.04-arm (Latest build is available at https://github.com/dotnet/core-setup#daily-builds)
dotnet-ubuntu.16.04-arm.1.2.0-beta-001291-00.tar.gz (Last updated on Jan 19)
dotnet-sdk-ubuntu.16.04-arm.1.0.0-preview5-004431.tar.gz
debian.8-armel
dotnet-debian.8-armel.1.2.0-beta-001271-00.tar.gz
tizen.4.0.0-armel
dotnet-tizen.4.0.0-armel.1.2.0-beta-001273-00.tar.gz

@hqueue hqueue added the arch-arm32 label Dec 5, 2016

@hseok-oh hseok-oh changed the title from Enabling Linux ARM32 to Enabling Linux ARM32 core-setup Dec 5, 2016

@hseok-oh hseok-oh added the os-linux label Dec 5, 2016

@gkhanna79

This comment has been minimized.

Member

gkhanna79 commented Dec 5, 2016

@janvorli

This comment has been minimized.

Member

janvorli commented Dec 6, 2016

As I have mentioned in the discussion in #724, I believe that we should be consistent in the way we do cross build over our repos. That means to use rootfs and to do the cross build using clang and not GCC. The same way we do it in CoreCLR and CoreFX.
I hope we can reuse the logic from the build.sh in CoreCLR here.

@gkhanna79

This comment has been minimized.

Member

gkhanna79 commented Dec 12, 2016

@hqueue @hseok-oh Is Ubuntu 14.04 an intended target for Arm32?

@hqueue

This comment has been minimized.

Member

hqueue commented Dec 13, 2016

@gkhanna79 AFAIK default rid is ubuntu.14.04-arm for coreclr ARM and default ARM rootfs is trusty which is ubuntu 14.04.

@hseok-oh

This comment has been minimized.

Contributor

hseok-oh commented Dec 13, 2016

@hqueue @gkhanna79 RaspberryPI2 used ubuntu 14.04. (but no longer maintained)
https://wiki.ubuntu.com/ARM/RaspberryPi

@gkhanna79

This comment has been minimized.

Member

gkhanna79 commented Dec 13, 2016

@hseok-oh If Ubuntu 14.04 is not maintained and supported, is there a specific reason to support it for Linux Arm32? If not, it will simplify bunch of things (e.g. libcoreclrtraceptprovider.so builds just fine for 16.04 but needs additional work for 14.04).

What do you think?

@gkhanna79

This comment has been minimized.

Member

gkhanna79 commented Dec 13, 2016

AFAIK default rid is ubuntu.14.04-arm for coreclr ARM and default ARM rootfs is trusty which is ubuntu 14.04

I don't know who chose those defaults or why. Could it be cause Tizen OS is based upon Ubuntu 14.04? do we build libcoreclrtraceptprovider.so for Tizen?

@gkhanna79

This comment has been minimized.

Member

gkhanna79 commented Dec 13, 2016

@kouvel This is the Linux Arm32 main work-list issue.

@gkhanna79

This comment has been minimized.

Member

gkhanna79 commented Dec 13, 2016

@hqueue

This comment has been minimized.

Member

hqueue commented Dec 14, 2016

AFAIK default rid is ubuntu.14.04-arm for coreclr ARM and default ARM rootfs is trusty which is ubuntu 14.04

I don't know who chose those defaults or why. Could it be cause Tizen OS is based upon Ubuntu 14.04? do we build libcoreclrtraceptprovider.so for Tizen?

@gkhanna79
I don't think it's related to Tizen OS.
The most possible explanation will be that ubuntu 14.04 is selected because of devices (e.g. ARM Emulator, Raspberry pi 2 and etc.) which were used to bring up CoreCLR ARM.
Please check out dotnet/coreclr#3805 for ARM emulator which is used at the very early stage of CoreCLR ARM32 bring up.

@gkhanna79 gkhanna79 changed the title from Enabling Linux ARM32 core-setup to Enabling Linux ARM32 for .NET Core Dec 15, 2016

@gkhanna79

This comment has been minimized.

Member

gkhanna79 commented Dec 15, 2016

Thanks for the explanation @hqueue. For now, we can continue to build for both 14.04 and 16.04 and see how far we will need the 14.04 implementation.

With regards to the emulator, I noticed that in CoreCLR repo, we bootup the emulator and setup the rootfs within it and then perform the build. This is different from the build instructions we have at https://github.com/dotnet/coreclr/blob/master/Documentation/building/cross-building.md where it comprises of two steps:

  1. Build rootfs for an architecture
  2. Perform a cross build of the repo

We strive hard to ensure that CI builds the same way as any developer would. Since the above steps are how any developer in the community would be expected to build for Arm32, what do you think of updating the CI script to do just that and not rely on emulator to be present for performing the build?

Secondly, the emulator should only be required to deploy binaries and execute tests. I notice that we only run 22 tests within the emulator, even though the repo has 11K+ tests. Do these 22 tests add any value from the CI perspective (e.g. here is a log of a recent PR that shows the 22 tests that ran - https://ci.dot.net/job/dotnet_coreclr/job/master/job/arm_emulator_cross_debug_ubuntu_prtest/1080/consoleFull) since they dont seem to cover key scenarios where breaks maybe introduced?

@hqueue

This comment has been minimized.

Member

hqueue commented Dec 16, 2016

@gkhanna79 For the first question, I will call @sjsinju @wateret, since I'm not aware of the latest of ARM CI.

@sjsinju @wateret Can you please answer the first question above regarding CI and ARM emulator ?

For the second one, I also think tests are not sufficient to cover key sccenarios. We will look into them.

@hqueue hqueue referenced this issue Dec 19, 2016

Merged

Add new rids for Linux ARM32 #712

4 of 4 tasks complete
@wateret

This comment has been minimized.

wateret commented Dec 20, 2016

@gkhanna79
CI cross-builds the same way as any developers would. You can check out arm32_ci_script.sh#L227. The emulator is only for running tests. Of course, we still need pre-built rootfs to cross-build though.

One more thing that may be confusing is that the CI has arm-softfp rootfs only which cannot be created by 'any developers'. There is my work on CoreFX for enabling arm-hardfp(ubuntu 14.04). After it's done I will apply it to CoreCLR as well.

@jyoungyun

This comment has been minimized.

Contributor

jyoungyun commented Dec 20, 2016

@gkhanna79 I couldn't find where libuv.so is built. Could you give me an advice?

@qmfrederik

This comment has been minimized.

Contributor

qmfrederik commented Dec 20, 2016

@jyoungyun not sure if it's the way libuv is built for .NET, but for raspberry pi the instructions so far have been to download it from the libuv project & compile it:

See the these comments from @leemgs related to dotnet/coreclr#6321

rpi2@arm# sudo apt-get install gyp
rpi2@arm# wget http://dist.libuv.org/dist/v1.0.0-rc1/libuv-v1.0.0-rc1.tar.gz 
(The latest version is v1.9.1: http://dist.libuv.org/dist/v1.9.1/libuv-v1.9.1.tar.gz)
rpi2@arm# tar -xvf libuv-v1.0.0-rc1.tar.gz
rpi2@arm# cd libuv-v1.0.0-rc1/
rpi2@arm# ./gyp_uv.py -f make -Duv_library=shared_library
rpi2@arm# make -C out
rpi2@arm# sudo cp out/Debug/lib.target/libuv.so /usr/lib/libuv.so.1.0.0-rc1
rpi2@arm# sudo ln -s libuv.so.1.0.0-rc1 /usr/lib/libuv.so.1

That worked for me to get Kestrel up & running on Raspberry Pi; but note that newer versions of libuv are now available at http://dist.libuv.org/dist/

@gkhanna79

This comment has been minimized.

Member

gkhanna79 commented Dec 20, 2016

CI cross-builds the same way as any developers would.

@wateret My point is that we should decouple cross building in CI from emulator. The nuance is that in CI, we setup rootfs from emulator and then perform cross-build while a regular developer will not. My suggestion is to fix the script to do the following:

  1. Without the emulator, perform the cross build (e.g. cross/build-rootfs.sh arm)
  2. Load the emulator only for running tests. Thus, if we were to decide that emulator may no longer be needed for testing for some reason, then cross-build continues to function as it is expected.
@gkhanna79

This comment has been minimized.

Member

gkhanna79 commented Dec 20, 2016

@hqueue did you find anything interesting about running the 22 tests in CI?

@ThadHouse

This comment has been minimized.

ThadHouse commented Mar 11, 2017

@olegsavelos the debian.8-armel build will not work on a raspberry pi zero. All of the builds here are for armv7 devices, with the different versions selecting between hard float ABI and soft float ABI. The Pi Zero (and the original Pi) is an armv6 device. I do not know what it will take to get armv6 working and building, but it could range from fairly easy (changing compiler flags) to very difficult. I just don't know.

@olegsavelos

This comment has been minimized.

olegsavelos commented Mar 12, 2017

Can anyone shed some light on Armv6 support ?

@JensNordenbro

This comment has been minimized.

JensNordenbro commented Mar 12, 2017

@jyoungyun , will the .net core tizen build work on an image including https://github.com/TizenTeam/meta-tizen on yocto?

@hqueue

This comment has been minimized.

Member

hqueue commented Mar 12, 2017

@olegsavelos If you meant PI Zero with Broadcom BCM2835, I'm afraid it's not easy to work dotnet-debian.8-armel.1.2.0-beta-001271-00.tar.gz on your Pi Zero. I also agree with @ThadHouse that PI Zero is based on armv6 and it could be range from easy to very difficult.

AFAIK current dotnet for ARM make use of Thumb-2 ISA which is not available in armv6 and vfp feature which is mostly available in armv7, but only optinally available in armv6. Armv6 support old Thumb not Thumb2, therefore you may have to build (1) all native component of dotnet for armv6 and also (2) may have to update JIT compiler and sub-component(written in assmebly language) of CoreCLR and CoreFX to make use of Thumb ISA and supported vfp feature of Broadcom BCM2835 instead of armv7. But I can't estimate how much effeort will be needed.

@olegsavelos

This comment has been minimized.

olegsavelos commented Mar 19, 2017

Mono seems to be working fine on armv6 so wouldn't it be possible to port the portions of code related to armv6 to .net core? In any case i think PI Zero is excellent platform for wide range of solutions and it would be shame not to support it.

@rahulreddy65

This comment has been minimized.

rahulreddy65 commented Mar 28, 2017

Can anyone tell me when will I be able to build and run programs on arm32 linux? thanks

@hqueue

This comment has been minimized.

Member

hqueue commented Mar 28, 2017

@rahulreddy65 If you are using Ubuntu 14.04 or Ubuntu 16.04 for arm, I think you can run C# programs on arm32 linux by installing binary from https://github.com/dotnet/core-setup#daily-builds.

If you want to build C# program from arm32 linux, it's not available yet. However you can build C# program from other environment, e.g. x64 linux or x64 Windows, and run the program on Ubuntu arm.

And there is another way you can just build and run C# program for arm32 linux with runtime framework version 2.0.0-beta-001620-00 and later. Please take a look at https://github.com/dotnet/core/blob/master/samples/RaspberryPiInstructions.md
However it also requires x64 environment to build the C# program.

@hqueue

This comment has been minimized.

Member

hqueue commented Mar 31, 2017

@olegsavelos I think you'd better open an issue about supporting armv6 at https://github.com/dotnet/coreclr, because most of required works are relevant to coreclr and experts who can answer your questions(e.g. supporting new CPU) are there too :)

@olegsavelos

This comment has been minimized.

olegsavelos commented Mar 31, 2017

@hqueue Thanks ! Will do that :)

@jyoungyun

This comment has been minimized.

Contributor

jyoungyun commented Mar 31, 2017

Finally, I got below results!!!

pi3@raspberry:~/Downloads/c$ time NUGET_PACKAGES=/home/pi3/Downloads/c/p dotnet build
Microsoft (R) Build Engine version 15.2.47.30403
Copyright (C) Microsoft Corporation. All rights reserved.

  c -> /home/pi3/Downloads/c/bin/Debug/netcoreapp2.0/c.dll

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:27.08

real	0m40.917s
user	0m43.890s
sys	0m1.390s

pi3@raspberry:~/Downloads/c$ time NUGET_PACKAGES=/home/pi3/Downloads/c/p dotnet run
Hello World!

real	0m51.861s
user	0m54.380s
sys	0m1.870s

The performance is very.... low, but Raspberry PI can build cs file itself!

@janvorli

This comment has been minimized.

Member

janvorli commented Mar 31, 2017

@jyoungyun are these results from release build of coreclr?

@jyoungyun

This comment has been minimized.

Contributor

jyoungyun commented Apr 6, 2017

@janvorli All of the nupkg used in the cli build got downloaded from myget server so this runtime(coreclr) is release build with -O1 option. If clang3.9 is enable, the result will be faster. :)

@janvorli

This comment has been minimized.

Member

janvorli commented Apr 6, 2017

@jyoungyun it was discovered in another issue that dotnet run command adds quite a lot of overhead to the start of the managed app (on x64 Linux, it adds about 2 seconds) compared to passing the managed assembly directly to dotnet without the "run".
Could you please try to measure that case as well? Just to be clear, I mean running time dotnet /home/pi3/Downloads/c/bin/Debug/netcoreapp2.0/c.dll

@jyoungyun

This comment has been minimized.

Contributor

jyoungyun commented Apr 6, 2017

@janvorli In raspberry PI3, passing dll as a directy argument is about 40 seconds faster than using dotnet run. This seems to be quite a lot of overhead in dotnet run command...

pi3@raspberry:~/Downloads/c$ time dotnet run
Hello World!

real	0m44.373s
user	0m56.000s
sys	0m1.800s

pi3@raspberry:~/Downloads/c$ time dotnet /home/pi3/Downloads/c/bin/Debug/netcoreapp2.0/c.dll
Hello World!

real	0m1.389s
user	0m1.340s
sys	0m0.040s
@janvorli

This comment has been minimized.

Member

janvorli commented Apr 6, 2017

@jyoungyun Thank you for measuring it! We really need to do something with it. I'll add this information to the other issue where the slowness was reported for x64.

@hqueue

This comment has been minimized.

Member

hqueue commented Apr 6, 2017

@janvorli @jyoungyun I think one of huge timz difference can be due to lack of native image for arm cli, which may be available for x64 cli itself.

@janvorli

This comment has been minimized.

Member

janvorli commented Apr 6, 2017

Well, even on x64, there is a huge difference in those two ways of launching the app. See dotnet/cli#6241

@nzain

This comment has been minimized.

nzain commented May 16, 2017

I see a lot of progress related to the Raspberry PI 2/3 and Ubuntu... but what is required to get my dotnet core 2 project running on an i.MX7 (Cortex-A7) with an angstrom based linux?

From my limited understanding the architecture should be armhf, which is the default architecture in all dotnet cross-compile guides I found so far.

I started cross-compiling CLR and FX, but it all seems centered around ubuntu. Will the result work on angstrom, as long as the required prerequisites (like libunwind, etc) are installed?

@JensNordenbro

This comment has been minimized.

JensNordenbro commented May 16, 2017

@nzain - try running using Linux (armhf) (for glibc based OS) in table on core-setup-readme.
This works fine for us (imx6 yocto poky) however you might have to bring in native dependencies into your os -dist manually.
Also notice that #2358 is not completely resolved however code is merged so it might just be that the fixes are not validated.

@zeroskyx

This comment has been minimized.

zeroskyx commented May 16, 2017

.NET Core 2.0 works flawlessly on a Raspberry Pi 3 with Raspbian Jessy Lite. The following is required to get it working:

On the vanilla installation of the OS run: sudo apt install libunwind8

In your .NET Core project include:

<TargetFramework>netcoreapp2.0</TargetFramework>
<RuntimeIdentifiers>win10-x64;linux-arm</RuntimeIdentifiers>

To publish on the command line/ vscode:

dotnet restore
dotnet build
dotnet publish --output Publish --runtime linux-arm

When your project name is MyProject: copy all the files in the Publish directory to the Pi and run

chmod 744 MyProject
./MyProject

Enjoy :)

@gkhanna79

This comment has been minimized.

Member

gkhanna79 commented May 31, 2017

@hqueue @jyoungyun Can we close this issue since the outstanding issue (per the list above) is in CoreCLR repo and tracked by dotnet/coreclr#8549?

@hqueue

This comment has been minimized.

Member

hqueue commented Jun 1, 2017

@gkhanna79 Sure. I will mark dotnet/coreclr#8549 in the list as done with comment.

@hqueue

This comment has been minimized.

Member

hqueue commented Jun 1, 2017

Closing this issue and remaining CoreCLR issues will be discussed at dotnet/coreclr#8549

@hqueue hqueue closed this Jun 1, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment