[NativeAOT-LLVM] [Question]: why is c++ 11 not installed in the root fs on the linux distros?

[yowl]

When using the docker image mcr.microsoft.com/dotnet-buildtools/prereqs:azurelinux-3.0-cross-amd64-net9.0 there is:

root [ /runtime ]# ls /crossrootfs/x64/usr/include/c++
5  5.4.0  v1
root [ /runtime ]# ls /usr/include/c++
13.2.0

This means that for example the stdc++ header <optional> is not available in the ROOTFS_DIR. We need it to compile the LLVM headers. Is it simply the case that up till now nothing has required c++ headers from a later version of c++, and if so any suggestions about the way forward?

Thanks.

[dicej]

FYI, I believe the directory names under e.g. /usr/include/c++ correspond to the compiler version (e.g. GCC 13.2.0 in the listing you provided above), which doesn't correspond to the C++ standard version (e.g. c++11, 14, 17, etc.). You'd need to cross-reference the compiler vendor and version with the C++ standard(s) it supports to get a complete picture.

[yowl]

Thanks, there is an optional under v1 but clang-18 is going for the /usr/include/c++/5/ directory, must be a way to make it use v1.

[yowl]

-isystem might be enough I guess.

[jkotas]

This cross-complication image is setup to target libraries (including standard C++ library) from Ubuntu 16. It looks like that standard C++ library is too old to compile LLVM.

@jkoritzinsky have been looking into options for how we can use the new standard C++ library for these cross-compilation images using static linking. @jkoritzinsky Do you have any suggestions here? (Note that this is experimental project, so it is ok to cut some corners here to make things work.)

[jkoritzinsky]

I think due to the issues we found when trying to consume libc++, I think we need a slightly different solution. The best solution would be to make a new image that contains a statically-linkable libc++ as we talked about for dotnet/runtime.

For now, I'd recommend using the "-sanitizer" image and passing the CxxStandardLibrary=libc++ property to the native components build. This would depend on LLVM being built with the same standard library.

Another alternative would be to manually link against the dotnet/llvm-project libc++ (if that's where llvm is coming from) with the same options that Mono uses.

[yowl]

Thanks, and I would need to install and build LLVM under the cross root (ROOTFS_DIR) ?

[jkoritzinsky]

You'd need to build LLVM against the rootfs, yes.

[yowl]

Thanks, bit of a follow up , _libc_fpstate is defined

struct _libc_fpstate
{
  unsigned long int __ctx(cw);
  unsigned long int __ctx(sw);
  unsigned long int __ctx(tag);
  unsigned long int __ctx(ipoff);
  unsigned long int __ctx(cssel);
  unsigned long int __ctx(dataoff);
  unsigned long int __ctx(datasel);
  struct _libc_fpreg _st[8];
  unsigned long int __ctx(status);
};

which doesnt have padding nor __glibc_reserved1 which

runtimelab/src/coreclr/pal/src/include/pal/context.h

Lines 356 to 360 in 53342be

	#if HAVE_FPSTATE_GLIBC_RESERVED1
	#define FPSTATE_RESERVED __glibc_reserved1
	#else
	#define FPSTATE_RESERVED padding
	#endif

doesn't like.

[yowl]

i.e.

In file included from /runtime/src/coreclr/pal/src/debug/debug.cpp:32:
  /runtime/src/coreclr/pal/src/include/pal/context.h:419:66: error: no member named 'padding' in '_libc_fpstate'
    419 |     return reinterpret_cast<_fpx_sw_bytes *>(&FPREG_Fpstate(uc)->FPSTATE_RESERVED[12]);
        |                                               ~~~~~~~~~~~~~~~~~  ^
  /runtime/src/coreclr/pal/src/include/pal/context.h:359:26: note: expanded from macro 'FPSTATE_RESERVED'
    359 | #define FPSTATE_RESERVED padding
        |                          ^
  1 error generated.

[jkotas]

As a quick workaround, you can try disabling XSTATE_SUPPORTED in src\coreclr\pal\src\CMakeLists.txt.

[yowl]

As a quick workaround, you can try disabling XSTATE_SUPPORTED in src\coreclr\pal\src\CMakeLists.txt.

Thanks, that works

[yowl]

Can I go to the "host" root, to get libraries, e.g. for

../../../bin/llvm-min-tblgen: error while loading shared libraries: libtinfo.so.5: cannot open shared object file: No such file or directory

This lib is not in the sysroot, can I assume the host arch is the same as the sysroot?

[jkoritzinsky]

If you link against the host, you can easily end up in a situation where the product becomes dependent on the host's capabilities, not the target. I wouldn't recommend it. I'd recommend installing more packages in the rootfs.

[yowl]

Thanks, it has apt-get installed, missed that.

[yowl]

I've tried various cmake options to "cross" compile LLVM on the sysroot, and make it build a llvm-min-tblgen that it can execute, but have not found a solution. Open to any suggestions that I will try.

[dicej]

@yowl do you have a branch somewhere with your latest attempt, plus maybe instructions for running the Docker image locally? I might have some time next week to experiment with it.

[yowl]

https://github.com/yowl/runtimelab/tree/linux-x64-jobs is the branch

[yowl]

docker run -it -v c:\github\runtimelab:/runtime   -w /runtime  mcr.microsoft.com/dotnet-buildtools/prereqs:azurelinux-3.0-cross-amd64-net9.0-sanitizer  bash

But change run to exec I think to start a new image, then go through the prequisites, install powershell (you should be able to use install-pwsh.sh for that which is basically

#!/usr/bin/env bash

mkdir -p $1

cd $1

curl -L -o powershell.tar.gz https://github.com/PowerShell/PowerShell/releases/download/v7.3.12/powershell-7.3.12-linux-x64.tar.gz

# Create the target folder where powershell will be placed
mkdir powershell7

# Expand powershell to the target folder
tar zxf powershell.tar.gz -C powershell7

# Set execute permissions
chmod +x powershell7/pwsh

Then try to run install-llvm.ps1 , cmake options are set on line 115.

pwsh eng/pipelines/runtimelab/install-llvm.ps1 -CloneDir wasm-tools -Configs release -CI

Add -NoClone to speed things up after LLVM is cloned.

[yowl]

All the prerequisites should be there with maybe the exception perhaps of libtinfo.so.5 which can be installed with apt-get from a chroot if it's not there.

[yowl]

I had some powershell syntax errors that caused cmake args to be merged. Fixing that has brought a new error

ld.lld: error: undefined symbol: __libc_csu_fini

This symbol is in the sysroot libc_nonshared.a, but the flags to clang++ are missing something or are in the wrong order.

/usr/local/bin/clang++ --sysroot=/crossrootfs/x64 -L/usr/lib -fPIC -fno-semantic-interposition \
-fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra \
-Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long \
-Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type \
-Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wmisleading-indentation \
-Wctad-maybe-unsupported -ffunction-sections -fdata-sections -O3 -DNDEBUG -fuse-ld=lld  \
-Wl,--gc-sections "CMakeFiles/llvm-min-tblgen.dir/TableGen.cpp.o" \
CMakeFiles/obj.LLVMTableGenCommon.dir/Attributes.cpp.o \
CMakeFiles/obj.LLVMTableGenCommon.dir/CodeGenIntrinsics.cpp.o \
CMakeFiles/obj.LLVMTableGenCommon.dir/DirectiveEmitter.cpp.o \
CMakeFiles/obj.LLVMTableGenCommon.dir/IntrinsicEmitter.cpp.o \
CMakeFiles/obj.LLVMTableGenCommon.dir/RISCVTargetDefEmitter.cpp.o \
CMakeFiles/obj.LLVMTableGenCommon.dir/SDNodeProperties.cpp.o \
CMakeFiles/obj.LLVMTableGenCommon.dir/VTEmitter.cpp.o -o ../../bin/llvm-min-tblgen  -Wl,-rpath,"\$ORIGIN/../lib" \
../../lib/libLLVMSupport.a ../../lib/libLLVMTableGen.a -lpthread ../../lib/libLLVMSupport.a -lrt -ldl -lpthread \
-lm /crossrootfs/x64/usr/lib/x86_64-linux-gnu/libz.so /crossrootfs/x64/usr/lib/x86_64-linux-gnu/libtinfo.so \
../../lib/libLLVMDemangle.a

[dicej]

@yowl Removing the "-DCMAKE_CXX_FLAGS=-L/usr/lib" part from https://github.com/yowl/runtimelab/blob/d20d26a4ab45c53351e3e5476a7a4e27bec44659/eng/pipelines/runtimelab/install-llvm.ps1#L116 fixed the __libc_csu_fini error for me, and LLVM built successfully.

[dicej]

bash build.sh clr.wasmjit+clr.aot+libs -c Debug failed, though:

clang : error : invalid linker name in argument '-fuse-ld=bfd' [/runtime/src/native/managed/cdacreader/src/cdacreader.csproj]

[yowl]

bash build.sh clr.wasmjit+clr.aot+libs -c Debug failed, though:

clang : error : invalid linker name in argument '-fuse-ld=bfd' [/runtime/src/native/managed/cdacreader/src/cdacreader.csproj]

In Discord SA pointed me to a missing /p:CrossBuild=true for the cause of picking bfd over lld

[dicej]

@yowl this commit worked for me: dicej@d97382f

[yowl]

@yowl Removing the "-DCMAKE_CXX_FLAGS=-L/usr/lib" part from https://github.com/yowl/runtimelab/blob/d20d26a4ab45c53351e3e5476a7a4e27bec44659/eng/pipelines/runtimelab/install-llvm.ps1#L116 fixed the __libc_csu_fini error for me, and LLVM built successfully.

Ugg, must have not tried that properly or not deleted the build folder first... Thanks.