Illegal Instruction running on ARMv7 SAMA5D31 processor despite cmakeargs DCLR_ARM_FPU_CAPABILITY=0x3 and DCLR_ARM_FPU_TYPE=vfpv3-d16 #32029

calderhayes · 2020-02-10T15:45:00Z

Hello dotnet team, I am having trouble with a cross compiled dotnet hello world application.

I am building on linux 64bit for an ARM processor (SAMA5D31) with the following information:

processor   : 0
model name  : ARMv7 Processor rev 1 (v7l)
BogoMIPS    : 351.43
Features    : half thumb fastmult vfp edsp thumbee vfpv3 vfpv3d16 tls vfpv4 
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x0
CPU part    : 0xc05
CPU revision    : 1

Hardware    : Atmel SAMA5
Revision    : 0000
Serial      : 0000000000000000

I keep receiving "Illegal Instruction" when running my code on the ARM machine.

I have been following the directions from "Build CoreCLR on Linux" (docker method), and the following issues:

#9969
dotnet/coreclr#27890
#546
#471

I have built both dotnet/coreclr (git hash a9f3fc16483eecfc47fb79c362811d870be02249 dotnet version 3.1) and dotnet/runtime (git hash 516956a dotnet version 5.0.0-dev).

My command to build the runtime for dotnet coreclr is:
sudo docker run --rm -v /media/calder/secondary/git/coreclr:/coreclr -w /coreclr -e ROOTFS_DIR=/crossrootfs/arm mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-cross-14.04-23cacb0-20190923200213 ./build.sh -arm -release -cross -stripsymbols -cmakeargs -DCLR_ARM_FPU_CAPABILITY=0x3 -cmakeargs -DCLR_ARM_FPU_TYPE=vfpv3-d16

My command to build the runtime for dotnet runtime is:
sudo docker run --rm -v /media/calder/secondary/git/runtime:/runtime -w /runtime -e ROOTFS_DIR=/crossrootfs/arm mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-cross-14.04-23cacb0-20191023143847 ./build.sh --arch arm --cross --cmakeargs -DCLR_ARM_FPU_CAPABILITY=0x3 --cmakeargs -DCLR_ARM_FPU_TYPE=vfpv3-d16 --clang9 --configuration Release

Not sure if this is relevant but out dotnet comment section is:

String dump of section '.comment':
  [     0]  GCC: (Ubuntu/Linaro 4.8.4-2ubuntu1~14.04.4) 4.8.4
  [    32]  clang version 9.0.1-svn374858-1~exp1~20191015044046.62 (branches/release_90)

My simple hello world application is made using my standard dotnet command to compile helloworld.

I have tried building just the byte code and using it with the runtime (this is using my usual linux64 dotnet compiler):
dotnet build -c Release

I then pass my bytecode in (bin/Release/netcoreapp3.1/) along with the compiled runtime to my ARM machine and run with:
~/dotnet-lib/dotnet helloworld.dll

I have tried this with both 3.1/5.0.0-dev.

I have also tried building a self contained executable:
dotnet publish --self-contained -r linux-arm

And then replacing dlls and .so files with my runtime libraries, but the same issue occurs.

Dependencies such as stdlibc++6 has been compiled for ARM as well.

I requested help from my colleague who is more experienced with embedded systems. He provided some stacktraces:

Program received signal SIGILL, Illegal instruction.
std::__detail::_Rehash_base<std::string, std::pair<std::string const, std::string>, std::allocator<std::pair<std::string const, std::string> >, std::__detail::_Select1st, std::equal_to<std::string>, std::hash<std::string>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::reserve (this=0x4246c8, __n=24)
  at /crossrootfs/arm/usr/lib/gcc/arm-linux-gnueabihf/4.8/../../../../include/c++/4.8/bits/hashtable_policy.h:819
819 /crossrootfs/arm/usr/lib/gcc/arm-linux-gnueabihf/4.8/../../../../include/c++/4.8/bits/hashtable_policy.h: No such file or directory.

#0  std::__detail::_Rehash_base<std::string, std::pair<std::string const, std::string>, std::allocator<std::pair<std::string const, std::string> >, std::__detail::_Select1st, std::equal_to<std::string>, std::hash<std::string>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::reserve (this=0x4246c8, __n=24)
    at /crossrootfs/arm/usr/lib/gcc/arm-linux-gnueabihf/4.8/../../../../include/c++/4.8/bits/hashtable_policy.h:819
#1  std::unordered_map<std::string, std::string, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::string> > >::reserve
    (this=0x4246c8, __n=24)
    at /crossrootfs/arm/usr/lib/gcc/arm-linux-gnueabihf/4.8/../../../../include/c++/4.8/bits/unordered_map.h:743
#2  coreclr_property_bag_t::coreclr_property_bag_t (this=0x4246c8)
    at /runtime/src/installer/corehost/cli/hostpolicy/coreclr.cpp:224
#3  0xb6bdbae0 in hostpolicy_context_t::hostpolicy_context_t (this=0x424698)
    at /runtime/src/installer/corehost/cli/hostpolicy/hostpolicy_context.h:15
#4  0xb6bd7ee4 in (anonymous namespace)::create_hostpolicy_context (
    hostpolicy_init=..., args=..., breadcrumbs_enabled=true)
    at /runtime/src/installer/corehost/cli/hostpolicy/hostpolicy.cpp:120
#5  0xb6bd7cf8 in corehost_main (argc=2, argv=<optimized out>)
    at /runtime/src/installer/corehost/cli/hostpolicy/hostpolicy.cpp:381
#6  0xb6c2a42e in execute_app (impl_dll_dir=..., init=0x424c98, 
    argc=<optimized out>, argv=<optimized out>)
---Type <return> to continue, or q <return> to quit---
    at /runtime/src/installer/corehost/cli/fxr/fx_muxer.cpp:145
#7  (anonymous namespace)::read_config_and_execute (host_command=..., host_info=..., 
    app_candidate=..., opts=..., new_argc=2, new_argv=0xbefffcb4, 
    mode=<optimized out>, out_buffer=<optimized out>, buffer_size=<optimized out>, 
    required_buffer_size=<optimized out>)
    at /runtime/src/installer/corehost/cli/fxr/fx_muxer.cpp:501
#8  fx_muxer_t::handle_exec_host_command (host_command=..., host_info=..., 
    app_candidate=..., opts=..., argc=2, argv=0xbefffcb4, argoff=1, mode=muxer, 
    result_buffer=0x0, buffer_size=0, required_buffer_size=0x0)
    at /runtime/src/installer/corehost/cli/fxr/fx_muxer.cpp:958
#9  0xb6c2909c in fx_muxer_t::execute (host_command=..., argc=2, argv=0xbefffcb4, 
    host_info=..., result_buffer=0x0, buffer_size=0, required_buffer_size=0x0)
    at /runtime/src/installer/corehost/cli/fxr/fx_muxer.cpp:547
#10 0xb6c24644 in hostfxr_main_startupinfo (argc=1103101952, argv=0x41c00000, 
    host_path=<optimized out>, dotnet_root=<optimized out>, 
    app_path=0x4230f4 "/home/dlmaint-sn/dotnet/dotnet.dll")
    at /runtime/src/installer/corehost/cli/fxr/hostfxr.cpp:33
#11 0x00409fa4 in exe_start (argc=2, argv=0xbefffcb4)
    at /runtime/src/installer/corehost/corehost.cpp:219
#12 0x0040a692 in main (argc=2, argv=0xbefffcb4)
    at /runtime/src/installer/corehost/corehost.cpp:286

And then this conversation:

0xb6bcdae6 <+122>: vcvt.f64.f32 d17, s18
11:32
Looks like it's trying to use D17 as a register which wouldn't exist on this FPU. So it is likely still the fpu issue.
11:35
__builtin_ceil(__n / max_load_factor())
That builtin function for ceil is causing it.
11:39
That's being called from coreclr_property_bag_t()
That call is made inside a stdc++ header file. The builtin itself is defined in the compiler. (edited)

So from what I can tell, one of two things could be happening:

I am missing something either in compilation steps or in dependencies.
There is some missed case of applying the vfpv3-d16 flag maybe?

Any assistance would be greatly appreciated.

Thanks,

Calder

The text was updated successfully, but these errors were encountered:

jkotas · 2020-02-10T16:33:07Z

CLR_ARM_FPU_CAPABILITY option is not wired into the installer partition of the build.

@franksinankaya @am11 Would you be interested in helping to get this fixed?

franksinankaya · 2020-02-10T20:46:02Z

@jkotas : can you provide some more details on what needs to be done?

jkotas · 2020-02-10T20:50:52Z

I would move this:

runtime/src/libraries/Native/Unix/CMakeLists.txt

Line 74 in 34f316e

add_compile_options(-mfpu=${CLR_ARM_FPU_TYPE})

runtime/src/coreclr/configurecompiler.cmake

Line 381 in fcfb4b8

add_compile_options(-mfpu=${CLR_ARM_FPU_TYPE})

to https://github.com/dotnet/runtime/blob/master/eng/native/configureplatform.cmake (?) so that it is used by everything in the project.

calderhayes · 2020-02-12T16:43:52Z

Hello dotnet team,

Thank you for the quick turn around on this issue. Unfortunately I still run into the Illegal Instruction.

I did a git pull of the master branch on dotnet/runtime, and am on the git hash of 8c10a98 .

I rebuilt the runtime using the following command (after deleting the artifacts folder):

sudo docker run --rm -v /media/calder/secondary/git/runtime:/runtime -w /runtime -e ROOTFS_DIR=/crossrootfs/arm mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-cross-14.04-23cacb0-20191023143847 ./build.sh --arch arm --cross --cmakeargs -DCLR_ARM_FPU_CAPABILITY=0x3 --cmakeargs -DCLR_ARM_FPU_TYPE=vfpv3-d16 --clang9 --configuration Release | tee compile.log

I send artifacts/packages/Release/Shipping/dotnet-runtime-5.0.0-dev-linux-arm.tar.gz to my device under a dotnet-lib folder, and then extract.

I also used this dotnet core-sdk release to build my hello world bytecode:

~/apps/dotnet-5.0.100-preview.1.20105.4/dotnet --info
.NET Core SDK (reflecting any global.json):
 Version:   5.0.100-preview.1.20105.4
 Commit:    b707d9e922

Runtime Environment:
 OS Name:     debian
 OS Version:  10
 OS Platform: Linux
 RID:         debian.10-x64
 Base Path:   /home/calder/apps/dotnet-5.0.100-preview.1.20105.4/sdk/5.0.100-preview.1.20105.4/

Host (useful for support):
  Version: 5.0.0-alpha.1.20103.10
  Commit:  a7f8cdc0de

.NET Core SDKs installed:
  5.0.100-preview.1.20105.4 [/home/calder/apps/dotnet-5.0.100-preview.1.20105.4/sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.App 5.0.0-alpha.1.20071.6 [/home/calder/apps/dotnet-5.0.100-preview.1.20105.4/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 5.0.0-alpha.1.20103.10 [/home/calder/apps/dotnet-5.0.100-preview.1.20105.4/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

I compile the bytecode using:

~/apps/dotnet-5.0.100-preview.1.20105.4/dotnet build -c Release

I then send the contents of bin/Release/netcoreapp5.0/* to my application into a helloworld folder.

On my device I run it like so:

~/helloworld$ ~/dotnet-lib/dotnet ./helloworld.dll
Illegal instruction

Here it is again, but with COREHOST_TRACE=1:

COREHOST_TRACE=1 ~/dotnet-lib/dotnet ./helloworld.dll
Tracing enabled @ Wed Feb 12 16:40:54 2020 GMT
--- Invoked dotnet [version: 5.0.0-dev, commit hash: 8c10a98c13263eea806d4fcecb76d29e38593539] main = {
/home/dlmaint-sn/dotnet-lib/dotnet
./helloworld.dll
}
Reading fx resolver directory=[/home/dlmaint-sn/dotnet-lib/host/fxr]
Considering fxr version=[5.0.0-dev]...
Detected latest fxr version=[/home/dlmaint-sn/dotnet-lib/host/fxr/5.0.0-dev]...
Resolved fxr [/home/dlmaint-sn/dotnet-lib/host/fxr/5.0.0-dev/libhostfxr.so]...
Invoking fx resolver [/home/dlmaint-sn/dotnet-lib/host/fxr/5.0.0-dev/libhostfxr.so] v2
Host path: [/home/dlmaint-sn/dotnet-lib/dotnet]
Dotnet path: [/home/dlmaint-sn/dotnet-lib/]
App path: [/home/dlmaint-sn/dotnet-lib/dotnet.dll]
Tracing enabled @ Wed Feb 12 16:40:54 2020 GMT
--- Invoked hostfxr_main_startupinfo [commit hash: 8c10a98c13263eea806d4fcecb76d29e38593539]
Checking if CoreCLR path exists=[/home/dlmaint-sn/dotnet-lib/libcoreclr.so]
--- Executing in muxer mode...
Using the provided arguments to determine the application to execute.
Using dotnet root path [/home/dlmaint-sn/dotnet-lib/]
App runtimeconfig.json from [/home/dlmaint-sn/helloworld/helloworld.dll]
Runtime config is cfg=/home/dlmaint-sn/helloworld/helloworld.runtimeconfig.json dev=/home/dlmaint-sn/helloworld/helloworld.runtimeconfig.dev.json
Attempting to read runtime config: /home/dlmaint-sn/helloworld/helloworld.runtimeconfig.json
Attempting to read dev runtime config: /home/dlmaint-sn/helloworld/helloworld.runtimeconfig.dev.json
Runtime config [/home/dlmaint-sn/helloworld/helloworld.runtimeconfig.json] is valid=[1]
--- The specified framework 'Microsoft.NETCore.App', version '5.0.0-alpha.1.20103.10', apply_patches=1, version_compatibility_range=minor is compatible with the previously referenced version '5.0.0-alpha.1.20103.10'.
--- Resolving FX directory, name 'Microsoft.NETCore.App' version '5.0.0-alpha.1.20103.10'
Multilevel lookup is true
Searching FX directory in [/home/dlmaint-sn/dotnet-lib]
Attempting FX roll forward starting from version='[5.0.0-alpha.1.20103.10]', apply_patches=1, version_compatibility_range=minor, roll_to_highest_version=0, prefer_release=0
'Roll forward' enabled with version_compatibility_range [minor]. Looking for the lowest release/pre-release greater than or equal version to [5.0.0-alpha.1.20103.10]
Found version [5.0.0-dev]
Framework reference resolved to version '5.0.0-dev'.
Changing Selected FX version from [] to [/home/dlmaint-sn/dotnet-lib/shared/Microsoft.NETCore.App/5.0.0-dev]
Chose FX version [/home/dlmaint-sn/dotnet-lib/shared/Microsoft.NETCore.App/5.0.0-dev]
Runtime config is cfg=/home/dlmaint-sn/dotnet-lib/shared/Microsoft.NETCore.App/5.0.0-dev/Microsoft.NETCore.App.runtimeconfig.json dev=/home/dlmaint-sn/dotnet-lib/shared/Microsoft.NETCore.App/5.0.0-dev/Microsoft.NETCore.App.runtimeconfig.dev.json
Attempting to read runtime config: /home/dlmaint-sn/dotnet-lib/shared/Microsoft.NETCore.App/5.0.0-dev/Microsoft.NETCore.App.runtimeconfig.json
Attempting to read dev runtime config: /home/dlmaint-sn/dotnet-lib/shared/Microsoft.NETCore.App/5.0.0-dev/Microsoft.NETCore.App.runtimeconfig.dev.json
Runtime config [/home/dlmaint-sn/dotnet-lib/shared/Microsoft.NETCore.App/5.0.0-dev/Microsoft.NETCore.App.runtimeconfig.json] is valid=[1]
--- Summary of all frameworks:
     framework:'Microsoft.NETCore.App', lowest requested version='5.0.0-alpha.1.20103.10', found version='5.0.0-dev', effective reference version='5.0.0-alpha.1.20103.10' apply_patches=1, version_compatibility_range=minor, roll_to_highest_version=0, folder=/home/dlmaint-sn/dotnet-lib/shared/Microsoft.NETCore.App/5.0.0-dev
Ignoring host interpreted additional probing path /home/calder/.dotnet/store/arm/netcoreapp5.0 as it does not exist.
Ignoring additional probing path /home/calder/.nuget/packages as it does not exist.
Executing as a framework-dependent app as per config file [/home/dlmaint-sn/helloworld/helloworld.runtimeconfig.json]
--- Resolving libhostpolicy.so version from deps json [/home/dlmaint-sn/dotnet-lib/shared/Microsoft.NETCore.App/5.0.0-dev/Microsoft.NETCore.App.deps.json]
Resolved version 5.0.0-dev from dependency manifest file [/home/dlmaint-sn/dotnet-lib/shared/Microsoft.NETCore.App/5.0.0-dev/Microsoft.NETCore.App.deps.json]
Directory core servicing at [] was not specified or found
Fallback directory core servicing at [opt/coreservicing] was not found
Did not find libhostpolicy.so in directory pkgs/runtime.linux-arm.Microsoft.NETCore.DotNetHostPolicy/5.0.0-dev/runtimes/linux-arm/native
The expected libhostpolicy.so directory is [/home/dlmaint-sn/dotnet-lib/shared/Microsoft.NETCore.App/5.0.0-dev]
Tracing enabled @ Wed Feb 12 16:40:54 2020 GMT
Reading from host interface version: [0x16041101:120] to initialize policy version: [0x16041101:120]
--- Invoked hostpolicy [commit hash: 8c10a98c13263eea806d4fcecb76d29e38593539] [runtime.linux-arm.Microsoft.NETCore.DotNetHostPolicy,5.0.0-dev,runtimes/linux-arm/native][arm] corehost_main = {
/home/dlmaint-sn/dotnet-lib/dotnet
./helloworld.dll
}
Deps file: 
Directory core servicing at [] was not specified or found
Fallback directory core servicing at [opt/coreservicing] was not found
Multilevel lookup is true
-- arguments_t: host_path='/home/dlmaint-sn/dotnet-lib/dotnet' app_root='/home/dlmaint-sn/helloworld/' deps='/home/dlmaint-sn/helloworld/helloworld.deps.json' core_svc='' mgd_app='/home/dlmaint-sn/helloworld/helloworld.dll'
-- arguments_t: dotnet shared store: '/home/dlmaint-sn/dotnet-lib/store/arm/netcoreapp5.0'
Illegal instruction

Is there something wrong with my method? Is there some other kind of information I can provide to help diagnose the issue?

Any help is greatly appreciated. Thanks!

Calder

am11 · 2020-02-12T17:11:15Z

@calderhayes, I was waiting for #32115 to go in so we get all changes in the master branch. Or you could switch to darc-master-23a166b8-333f-468f-968d-5544e961b5c9 branch, merge master into it and try the build.

calderhayes · 2020-02-12T18:00:18Z

@am11 that worked!!

Thank you all for your quick assistance!

Calder

calderhayes · 2020-02-14T16:50:52Z

FYI I also confirmed this on latest master (git hash c4683fb ).

Thanks again.

Dotnet-GitSync-Bot added the untriaged New issue has not been triaged by the area owner label Feb 10, 2020

jkotas added arch-arm32 os-linux Linux OS (any supported distro) labels Feb 10, 2020

am11 mentioned this issue Feb 10, 2020

Make FPU settings configurable for installer #32053

Merged

jeffschwMSFT added the area-Meta label Feb 11, 2020

calderhayes closed this as completed Feb 14, 2020

ghost locked as resolved and limited conversation to collaborators Dec 10, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Illegal Instruction running on ARMv7 SAMA5D31 processor despite cmakeargs DCLR_ARM_FPU_CAPABILITY=0x3 and DCLR_ARM_FPU_TYPE=vfpv3-d16 #32029

Illegal Instruction running on ARMv7 SAMA5D31 processor despite cmakeargs DCLR_ARM_FPU_CAPABILITY=0x3 and DCLR_ARM_FPU_TYPE=vfpv3-d16 #32029

calderhayes commented Feb 10, 2020

jkotas commented Feb 10, 2020

franksinankaya commented Feb 10, 2020

jkotas commented Feb 10, 2020

calderhayes commented Feb 12, 2020

am11 commented Feb 12, 2020

calderhayes commented Feb 12, 2020

calderhayes commented Feb 14, 2020

Illegal Instruction running on ARMv7 SAMA5D31 processor despite cmakeargs DCLR_ARM_FPU_CAPABILITY=0x3 and DCLR_ARM_FPU_TYPE=vfpv3-d16 #32029

Illegal Instruction running on ARMv7 SAMA5D31 processor despite cmakeargs DCLR_ARM_FPU_CAPABILITY=0x3 and DCLR_ARM_FPU_TYPE=vfpv3-d16 #32029

Comments

calderhayes commented Feb 10, 2020

jkotas commented Feb 10, 2020

franksinankaya commented Feb 10, 2020

jkotas commented Feb 10, 2020

calderhayes commented Feb 12, 2020

am11 commented Feb 12, 2020

calderhayes commented Feb 12, 2020

calderhayes commented Feb 14, 2020