Arm64 official builds broken

[mmitche]

@janvorli @sdmaclea The arm64 builds appear to still be broken, though the errors is different than before. Can you take a look?

2018-03-12T21:11:49.6735353Z [command]/usr/bin/docker run --rm -e ROOTFS_DIR --name coreclr-cross-1468025 --ulimit core=-1 -v coreclr-cross-1468025:/root/coreclr -w=/root/coreclr microsoft/dotnet-buildtools-prereqs:ubuntu-16.04-cross-arm64-d809634-20180312171656 ./build.sh Release arm64 skipnuget cross -skiprestore stripSymbols -OfficialBuildId=20180312-08 -- /flp:v=diag
2018-03-12T21:11:50.3487530Z Commencing CoreCLR Repo build
2018-03-12T21:11:50.3810123Z WARNING: Can not determine runtime id for current distro.
2018-03-12T21:11:50.3826831Z Setting up directories for build
2018-03-12T21:11:50.3932484Z Checking prerequisites...
2018-03-12T21:11:50.3947447Z Restoring the OptimizationData package
2018-03-12T21:11:50.4016998Z Running init-tools.sh
2018-03-12T21:11:50.4107529Z Tools are already initialized
2018-03-12T21:11:50.4177439Z Running: /root/coreclr/Tools/dotnetcli/dotnet /root/coreclr/Tools/run.exe /root/coreclr/config.json sync -optdata
2018-03-12T21:11:50.5888629Z Running: /root/coreclr/Tools/msbuild.sh /nologo /verbosity:minimal /clp:Summary  /flp:v=detailed;LogFile=sync.log /t:RestoreOptData /p:RestoreDuringBuild=true  ./build.proj 
2018-03-12T21:11:50.6988000Z 
2018-03-12T21:11:50.7007456Z Welcome to .NET Core!
2018-03-12T21:11:50.7023001Z ---------------------
2018-03-12T21:11:50.7042280Z Learn more about .NET Core @ https://aka.ms/dotnet-docs. Use dotnet --help to see available commands or go to https://aka.ms/dotnet-cli-docs.
2018-03-12T21:11:50.7052493Z 
2018-03-12T21:11:50.7066354Z Telemetry
2018-03-12T21:11:50.7083770Z --------------
2018-03-12T21:11:50.7102842Z The .NET Core tools collect usage data in order to improve your experience. The data is anonymous and does not include command-line arguments. The data is collected by Microsoft and shared with the community.
2018-03-12T21:11:50.7121282Z You can opt out of telemetry by setting a DOTNET_CLI_TELEMETRY_OPTOUT environment variable to 1 using your favorite shell.
2018-03-12T21:11:50.7138466Z You can read more about .NET Core tools telemetry @ https://aka.ms/dotnet-cli-telemetry.
2018-03-12T21:11:54.9024262Z 
2018-03-12T21:11:54.9045627Z Build succeeded.
2018-03-12T21:11:54.9061524Z     0 Warning(s)
2018-03-12T21:11:54.9076579Z     0 Error(s)
2018-03-12T21:11:54.9083704Z 
2018-03-12T21:11:54.9098116Z Time Elapsed 00:00:03.97
2018-03-12T21:11:54.9574926Z Command execution succeeded.
2018-03-12T21:11:54.9609979Z Command successfully completed.
2018-03-12T21:11:56.6570179Z Laying out dynamically generated files consumed by the build system 
2018-03-12T21:11:56.6585975Z Laying out dynamically generated Event test files, etmdummy stub functions, and external linkages
2018-03-12T21:11:57.3787712Z  Generating Dummy Event Headers
2018-03-12T21:11:57.3806122Z  Generating Event Headers
2018-03-12T21:11:57.3819491Z Generating Event Logging Tests
2018-03-12T21:11:57.4115963Z Laying out dynamically generated EventPipe Implementation
2018-03-12T21:11:58.4131740Z Laying out dynamically generated Event Logging Implementation of Lttng
2018-03-12T21:11:58.7453935Z Commencing build of CoreCLR component for Linux.arm64.Release in /root/coreclr/bin/obj/Linux.arm64.Release
2018-03-12T21:11:58.7472108Z /root/coreclr
2018-03-12T21:11:58.7498223Z Running init-tools.sh
2018-03-12T21:11:58.7563244Z Tools are already initialized
2018-03-12T21:11:58.7581386Z Running: /root/coreclr/Tools/dotnetcli/dotnet /root/coreclr/Tools/run.exe /root/coreclr/config.json build -Project=/root/coreclr/build.proj -generateHeaderUnix -NativeVersionSourceFile=/root/coreclr/bin/obj/Linux.arm64.Release/version.cpp -MsBuildEventLogging=/l:BinClashLogger,Tools/Microsoft.DotNet.Build.Tasks.dll;LogFile=binclash.log -BuildArch=arm64 -BuildType=Release -BuildOS=Linux -skiprestore -OfficialBuildId=20180312-08 -- /flp:v=diag
2018-03-12T21:11:58.9238650Z Running: /root/coreclr/Tools/msbuild.sh /nologo /verbosity:minimal /clp:Summary  /l:BinClashLogger,Tools/Microsoft.DotNet.Build.Tasks.dll;LogFile=binclash.log /p:RestoreDuringBuild=false  /root/coreclr/build.proj /p:__BuildType=Release /p:__BuildArch=arm64 /p:__BuildOS=Linux  /t:GenerateVersionSourceFile /p:GenerateVersionSourceFile=true  /maxcpucount /p:NativeVersionSourceFile=/root/coreclr/bin/obj/Linux.arm64.Release/version.cpp /p:OfficialBuildId=20180312-08 /p:RestoreDefaultOptimizationDataPackage=false /p:UsePartialNGENOptimization=false /p:PortableBuild=true       /flp:v=diag
2018-03-12T21:11:59.7509022Z 
2018-03-12T21:11:59.7532301Z Build succeeded.
2018-03-12T21:11:59.7550000Z     0 Warning(s)
2018-03-12T21:11:59.7564829Z     0 Error(s)
2018-03-12T21:11:59.7571541Z 
2018-03-12T21:11:59.7585089Z Time Elapsed 00:00:00.43
2018-03-12T21:12:00.6591786Z Command execution succeeded.
2018-03-12T21:12:00.6605896Z Command successfully completed.
2018-03-12T21:12:00.6619487Z ~/coreclr/bin/obj/Linux.arm64.Release ~/coreclr
2018-03-12T21:12:00.6636769Z Invoking "/root/coreclr/src/pal/tools/gen-buildsys-clang.sh" "/root/coreclr" 3 9 arm64 Release  Include_Tests  -DCLR_CMAKE_TARGET_OS=Linux -DCLR_CMAKE_PACKAGES_DIR=/root/coreclr/packages -DCLR_CMAKE_PGO_INSTRUMENT=0 -DCLR_CMAKE_OPTDATA_VERSION=99.99.99-master-20180228-0037 -DCLR_CMAKE_PGO_OPTIMIZE=1  -DSTRIP_SYMBOLS=true
2018-03-12T21:12:00.6654269Z Including tests directory in build.
2018-03-12T21:12:00.6666780Z loading initial cache file /root/coreclr/cross/arm64/tryrun.cmake
2018-03-12T21:12:00.6681325Z -- The C compiler identification is Clang 3.9.1
2018-03-12T21:12:00.6698519Z -- The CXX compiler identification is Clang 3.9.1
2018-03-12T21:12:00.6714328Z -- Check for working C compiler: /usr/bin/clang-3.9
2018-03-12T21:12:00.6741467Z -- Check for working C compiler: /usr/bin/clang-3.9 -- broken
2018-03-12T21:12:00.6762388Z CMake Error at /usr/share/cmake-3.5/Modules/CMakeTestCCompiler.cmake:61 (message):
2018-03-12T21:12:00.6816230Z   The C compiler "/usr/bin/clang-3.9" is not able to compile a simple test
2018-03-12T21:12:00.6830396Z   program.
2018-03-12T21:12:00.6836673Z 
2018-03-12T21:12:00.6850334Z   It fails with the following output:
2018-03-12T21:12:00.6856902Z 
2018-03-12T21:12:00.6869312Z    Change Dir: /root/coreclr/bin/obj/Linux.arm64.Release/CMakeFiles/CMakeTmp
2018-03-12T21:12:00.6875727Z 
2018-03-12T21:12:00.6889812Z   
2018-03-12T21:12:00.6895963Z 
2018-03-12T21:12:00.6909081Z   Run Build Command:"/usr/bin/make" "cmTC_b4963/fast"
2018-03-12T21:12:00.6915407Z 
2018-03-12T21:12:00.6931494Z   /usr/bin/make -f CMakeFiles/cmTC_b4963.dir/build.make
2018-03-12T21:12:00.6945995Z   CMakeFiles/cmTC_b4963.dir/build
2018-03-12T21:12:00.6952275Z 
2018-03-12T21:12:00.6965837Z   make[1]: Entering directory
2018-03-12T21:12:00.6979975Z   '/root/coreclr/bin/obj/Linux.arm64.Release/CMakeFiles/CMakeTmp'
2018-03-12T21:12:00.6988267Z 
2018-03-12T21:12:00.7000811Z   Building C object CMakeFiles/cmTC_b4963.dir/testCCompiler.c.o
2018-03-12T21:12:00.7007394Z 
2018-03-12T21:12:00.7021534Z   /usr/bin/clang-3.9 -Wall -std=c11 -target aarch64-linux-gnu
2018-03-12T21:12:00.7039607Z   --sysroot=/crossrootfs/arm64 -o CMakeFiles/cmTC_b4963.dir/testCCompiler.c.o
2018-03-12T21:12:00.7055844Z   -c
2018-03-12T21:12:00.7070351Z   /root/coreclr/bin/obj/Linux.arm64.Release/CMakeFiles/CMakeTmp/testCCompiler.c
2018-03-12T21:12:00.7141890Z 
2018-03-12T21:12:00.7149002Z 
2018-03-12T21:12:00.7163530Z   Linking C executable cmTC_b4963
2018-03-12T21:12:00.7170815Z 
2018-03-12T21:12:00.7186997Z   /usr/bin/cmake -E cmake_link_script CMakeFiles/cmTC_b4963.dir/link.txt
2018-03-12T21:12:00.7205974Z   --verbose=1
2018-03-12T21:12:00.7214861Z 
2018-03-12T21:12:00.7231607Z   /usr/bin/clang-3.9 -Wall -std=c11 -target aarch64-linux-gnu -B
2018-03-12T21:12:00.7273466Z   /crossrootfs/arm64/usr/lib/gcc/aarch64-linux-gnu
2018-03-12T21:12:00.7291564Z   -L/crossrootfs/arm64/lib/aarch64-linux-gnu --sysroot=/crossrootfs/arm64
2018-03-12T21:12:00.7307805Z   CMakeFiles/cmTC_b4963.dir/testCCompiler.c.o -o cmTC_b4963 -rdynamic
2018-03-12T21:12:00.7316199Z 
2018-03-12T21:12:00.7331189Z   /usr/bin/aarch64-linux-gnu-ld: cannot find crt1.o: No such file or
2018-03-12T21:12:00.7345767Z   directory
2018-03-12T21:12:00.7353346Z 
2018-03-12T21:12:00.7386226Z   /usr/bin/aarch64-linux-gnu-ld: cannot find crti.o: No such file or
2018-03-12T21:12:00.7411598Z   directory
2018-03-12T21:12:00.7418229Z 
2018-03-12T21:12:00.7432514Z   /usr/bin/aarch64-linux-gnu-ld: cannot find crtbegin.o: No such file or
2018-03-12T21:12:00.7447497Z   directory
2018-03-12T21:12:00.7453666Z 
2018-03-12T21:12:00.7467659Z   /usr/bin/aarch64-linux-gnu-ld: cannot find -lgcc
2018-03-12T21:12:00.7475705Z 
2018-03-12T21:12:00.7491265Z   /usr/bin/aarch64-linux-gnu-ld: cannot find -lgcc_s
2018-03-12T21:12:00.7499338Z 
2018-03-12T21:12:00.7515024Z   /usr/bin/aarch64-linux-gnu-ld: cannot find -lc
2018-03-12T21:12:00.7523951Z 
2018-03-12T21:12:00.7539301Z   /usr/bin/aarch64-linux-gnu-ld: cannot find -lgcc
2018-03-12T21:12:00.7547740Z 
2018-03-12T21:12:00.7589156Z   /usr/bin/aarch64-linux-gnu-ld: cannot find -lgcc_s
2018-03-12T21:12:00.7598233Z 
2018-03-12T21:12:00.7613061Z   /usr/bin/aarch64-linux-gnu-ld: cannot find crtend.o: No such file or
2018-03-12T21:12:00.7627354Z   directory
2018-03-12T21:12:00.7633522Z 
2018-03-12T21:12:00.7649216Z   /usr/bin/aarch64-linux-gnu-ld: cannot find crtn.o: No such file or
2018-03-12T21:12:00.7665097Z   directory
2018-03-12T21:12:00.7671299Z 
2018-03-12T21:12:00.7686104Z   clang: error: linker command failed with exit code 1 (use -v to see
2018-03-12T21:12:00.7700206Z   invocation)
2018-03-12T21:12:00.7706474Z 
2018-03-12T21:12:00.7720940Z   CMakeFiles/cmTC_b4963.dir/build.make:97: recipe for target 'cmTC_b4963'
2018-03-12T21:12:00.7735493Z   failed
2018-03-12T21:12:00.7741647Z 
2018-03-12T21:12:00.7753858Z   make[1]: *** [cmTC_b4963] Error 1
2018-03-12T21:12:00.7763651Z 
2018-03-12T21:12:00.7775815Z   make[1]: Leaving directory
2018-03-12T21:12:00.7790449Z   '/root/coreclr/bin/obj/Linux.arm64.Release/CMakeFiles/CMakeTmp'
2018-03-12T21:12:00.7798615Z 
2018-03-12T21:12:00.7813327Z   Makefile:126: recipe for target 'cmTC_b4963/fast' failed
2018-03-12T21:12:00.7824062Z 
2018-03-12T21:12:00.7836223Z   make: *** [cmTC_b4963/fast] Error 2
2018-03-12T21:12:00.7842812Z 
2018-03-12T21:12:00.7855232Z   
2018-03-12T21:12:00.7861369Z 
2018-03-12T21:12:00.7873473Z   
2018-03-12T21:12:00.7879609Z 
2018-03-12T21:12:00.7899173Z   CMake will not be able to correctly generate this project.
2018-03-12T21:12:00.7912423Z Call Stack (most recent call first):
2018-03-12T21:12:00.7925407Z   CMakeLists.txt:9 (project)
2018-03-12T21:12:00.7931644Z 
2018-03-12T21:12:00.7937779Z 
2018-03-12T21:12:00.7951773Z -- Configuring incomplete, errors occurred!
2018-03-12T21:12:00.7966645Z See also "/root/coreclr/bin/obj/Linux.arm64.Release/CMakeFiles/CMakeOutput.log".
2018-03-12T21:12:00.7981542Z See also "/root/coreclr/bin/obj/Linux.arm64.Release/CMakeFiles/CMakeError.log".
2018-03-12T21:12:00.7994437Z ~/coreclr
2018-03-12T21:12:00.8007438Z Failed to generate CoreCLR component build project!

https://devdiv.visualstudio.com/DefaultCollection/DevDiv/_build?_a=summary&buildId=1468025

[sdmaclea]

The failing command is something like this
/usr/bin/clang-3.9 -Wall -std=c11 -target aarch64-linux-gnu -B /crossrootfs/arm64/usr/lib/gcc/aarch64-linux-gnu -L/crossrootfs/arm64/lib/aarch64-linux-gnu --sysroot=/crossrootfs/arm64 hw.c -o hw.arm64 -rdynamic

I get similar errors when the rootfs is not present at /crossrootfs/arm64

/usr/bin/aarch64-linux-gnu-ld: cannot find crt1.o: No such file or directory
/usr/bin/aarch64-linux-gnu-ld: cannot find crti.o: No such file or directory
/usr/bin/aarch64-linux-gnu-ld: cannot find crtbegin.o: No such file or directory
/usr/bin/aarch64-linux-gnu-ld: cannot find -lgcc
/usr/bin/aarch64-linux-gnu-ld: cannot find -lgcc_s
/usr/bin/aarch64-linux-gnu-ld: cannot find -lc
/usr/bin/aarch64-linux-gnu-ld: cannot find -lgcc
/usr/bin/aarch64-linux-gnu-ld: cannot find -lgcc_s
/usr/bin/aarch64-linux-gnu-ld: cannot find crtend.o: No such file or directory
/usr/bin/aarch64-linux-gnu-ld: cannot find crtn.o: No such file or directory

I'll try to figure out how to load the docker tag and see if I can see what is going on

[sdmaclea]

From the docker build logs https://ci.dot.net/job/dotnet_dotnet-buildtools-prereqs-docker/view/Official%20Builds/job/master/job/ubuntu_16_prtest/11/console

18:06:46 I: Extracting zlib1g...
18:06:46 I: Running command: chroot /rootfs/arm64 /debootstrap/debootstrap --second-stage
18:06:46 chroot: failed to run command '/debootstrap/debootstrap': Exec format error
18:06:46 /scripts/cross/build-rootfs.sh: line 138: install_Failed: command not found
18:06:46 chroot: failed to run command 'apt-get': No such file or directory
18:06:46 /scripts/cross/build-rootfs.sh: line 146: install_Failed: command not found
18:06:46 chroot: failed to run command 'apt-get': No such file or directory
18:06:46 /scripts/cross/build-rootfs.sh: line 150: install_Failed: command not found
18:06:46 chroot: failed to run command 'apt-get': No such file or directory
18:06:46 /scripts/cross/build-rootfs.sh: line 154: install_Failed: command not found
18:06:46 chroot: failed to run command 'symlinks': No such file or directory
18:06:46 /scripts/cross/build-rootfs.sh: line 158: install_Failed: command not found
18:06:47 -- EXECUTING: docker build -t microsoft/dotnet-buildtools-prereqs:ubuntu-16.04-cross-arm64-2dd1363-20180312004842 -f src/ubuntu/16.04/cross/arm64/Dockerfile src/ubuntu/16.04/cross/arm64

The rootfs build failed.

[sdmaclea]

Looks like "arm" and "arm64" rootfs images are failed to build in every log.

[janvorli]

I can see many failures to mount binfmt_misc filesystem in the log. I wonder if that's related to the issue.

18:03:27 mount: permission denied
18:03:27 update-binfmts: warning: Couldn't mount the binfmt_misc filesystem on /proc/sys/fs/binfmt_misc.

[mmitche]

@sdmaclea Do you know what dotnet-buildtools-prereqs-docker sha generated a correct image? We may be able to rerun and diff the logs.

[sdmaclea]

The current arm pipeline build is using ubuntu-14.04-cross-0cd4667-20170319080304. That should correspond to a sha of 0cd4667. Unfortunately it is 10 months before the initial commit in the public repository.

The earliest logs are from build 5 which failed on both "arm" and "arm64" on Unbuntu 16.04. I'll take a look at the 14.04 builds.

[sdmaclea]

The earliest 14.04 log has the same issue. https://ci.dot.net/job/dotnet_dotnet-buildtools-prereqs-docker/view/Official%20Builds/job/master/job/ubuntu_14_prtest/5/console

15:59:10 I: Extracting zlib1g...
15:59:10 I: Running command: chroot /rootfs/arm /debootstrap/debootstrap --second-stage
15:59:10 chroot: failed to run command '/debootstrap/debootstrap': Exec format error
15:59:10 /scripts/cross/build-rootfs.sh: line 138: install_Failed: command not found
15:59:10 chroot: failed to run command 'apt-get': No such file or directory
15:59:10 /scripts/cross/build-rootfs.sh: line 146: install_Failed: command not found
15:59:10 chroot: failed to run command 'apt-get': No such file or directory
15:59:10 /scripts/cross/build-rootfs.sh: line 150: install_Failed: command not found
15:59:10 chroot: failed to run command 'apt-get': No such file or directory
15:59:10 /scripts/cross/build-rootfs.sh: line 154: install_Failed: command not found
15:59:10 chroot: failed to run command 'symlinks': No such file or directory
15:59:10 /scripts/cross/build-rootfs.sh: line 158: install_Failed: command not found
15:59:10 /rootfs/arm /
15:59:10 can't find file to patch at input line 4
15:59:10 Perhaps you used the wrong -p or --strip option?
15:59:10 The text leading up to this was:

[sdmaclea]

Looks like @janvorli is right those binfmt errors are problematic.

http://neophob.com/2014/10/run-arm-binaries-in-your-docker-container-using-boot2docker/

There was also hint elsewhere that configuring the host to mount binfmt may also fix the issue

[jashook]

@RussKeldorph changing the milestone to release/2.1 because this will affect arm build as well.