Specify the correct cross-compiling toochain for aarch64 in the docs

[michpappas]

Feature request summary

This one is related to future changes in the docs: https://github.com/unikraft/unikraft/tree/staging/doc/guides
Specifically, when adding instructions for cross-compiling unikraft for AArch64, make sure to specify the bare-metal toolchain variant (aarch66-none-elf), as it is common for people to use a linux toolchain (eg aarch-none-linux) and end up having problems with the build.

Describe alternatives

No response

Related architectures

Arm, AArch64

Related platforms

No response

Additional context

The cross-compiling toolchains provided by Arm come in different variants (arm-none-eabi, arm-none-linux-gnueabihf, aarch64-none-elf, aarch64-none-linux). Choosing the wrong one can cause problems.

Example: libgcc of gcc >= 10.1 expects that the __getauxval symbol is provided by glibc. Trying to build Unikraft with aarch64-none-linux will result into gcc emmitting the following error:

/home/mpp/toolchains/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu/bin/../lib/gcc/aarch64-none-linux-gnu/10.3.1/../../../../aarch64-none-linux-gnu/bin/ld: /tmp/uk_event/app-helloworld/build/app-helloworld_kvm-arm64.o: in function `init_have_lse_atomics':
/data/jenkins/workspace/GNU-toolchain/arm-10/src/gcc/libgcc/config/aarch64/lse-init.c:44: undefined reference to `__getauxval'
/data/jenkins/workspace/GNU-toolchain/arm-10/src/gcc/libgcc/config/aarch64/lse-init.c:44:(.text.startup+0x4c): relocation truncated to fit: R_AARCH64_CALL26 against undefined symbol `__getauxval'
collect2: error: ld returned 1 exit status
make[2]: *** [/tmp/uk_event/unikraft/plat/kvm/Linker.uk:24: /tmp/uk_event/app-helloworld/build/app-helloworld_kvm-arm64.dbg] Error 1
make[1]: *** [Makefile:990: sub-make] Error 2
make[1]: Leaving directory '/tmp/uk_event/unikraft'
make: *** [Makefile:8: all] Error 2

Although this issue could be easily fixed by adding -wno-outline-atomics to ARCHFLAGS, this would not be the correct way to address the issue. Instead, one should be using the baremetal variant (aarch64-none-elf) for building Unikraft.

[kubanrob]

Static EmuTLS implementation for aarch64-none-elf

Due to the reasons documented here, we would like to use the bare-metal toolchain (aarch64-none-elf) instead of the Linux one (aarch64-none-linux-gnu). Unfortunately, in that toolchain, gcc is configured with --disable-tls, which make it use emutls insteads generating the instructions to access the TLS through the tpidr_el0 register.

Basically all accesses to the TLS are replace with a call to __emutls_get_address, which is passed a description of the object in TLS. You can see the symbols for the object description and templates for initialization in the debug image.

$objdump -t build/app-pte-test-private_kvm-arm64.dbg | grep emu
0000000040194218 l     O .rodata        0000000000000008 __emutls_t.pthread_self
00000000401d3318 l     O .data  0000000000000020 __emutls_v.pthread_self
0000000040194220 l     O .rodata        0000000000000010 __emutls_t.cl_status
0000000040194230 l     O .rodata        0000000000000004 __emutls_t.cl_uktls_magic
00000000401d3338 l     O .data  0000000000000020 __emutls_v.cl_status
00000000401d3358 l     O .data  0000000000000020 __emutls_v.cl_uktls_magic
00000000401d3388 l     O .data  0000000000000020 __emutls_v.child_tid
0000000040196720 l     O .rodata        0000000000000008 __emutls_t._uk_syscall_return_addr
00000000401d36d0 l     O .data  0000000000000020 __emutls_v.entry.7769
00000000401d36f0 l     O .data  0000000000000020 __emutls_v.result.7770
0000000000000000 l    df *ABS*  0000000000000000 emutls.c
0000000040190f28 l     F .text  0000000000000040 .hidden __emutls_register_common
00000000401d33a8 l     O .data  0000000000000020 __emutls_v._uk_syscall_return_addr
0000000040190e70 l     F .text  00000000000000b8 .hidden __emutls_get_address

Unfortunately the libgcc implementation uses dynamic memory allocation and thread keys, and brakes Unikraft assumptions about the TLS layout.

Fortunately, overriding the __emutls_get_address is as simple as providing the symbol yourself.

First idea for implementation:

• find a way to enumerate __emutls_v.* symbols on runtime
• put the symbol on deterministic order (eg. alignment + name)
• from that order, deterministically derive static TLS layout used in the implementation of libcontext
• put put offsets into __emutls_v. descriptors on (first) TLS initialization
• than use the offset in __emutls_get_address