arm64 compat vdso hard codes bfd as linker

[nickdesaulniers]

cc @fvincenzo
I just notices that arch/arm64/kernel/vdso32/Makefile has:

124 VDSO_LDFLAGS += $(call cc32-ldoption,-fuse-ld=bfd)                                                                             

Not sure if LLD doesn't work here?

[nathanchance]

I am not sure that was intentional, the comment mentions that it was copied from the ARM vDSO, where that was a flag before commit eliminated.

This works for me, I'll try to get some more testing on it from 0day and my Raspberry Pi then ship it along.

From 4293c3846175bbf30d0088b340a981516f487fbb Mon Sep 17 00:00:00 2001
From: Nathan Chancellor <natechancellor@gmail.com>
Date: Thu, 21 May 2020 21:03:26 -0700
Subject: [PATCH] arm64: vdso32: Allow ld.lld to properly link the VDSO

As it stands now, the vdso32 Makefile hardcodes the linker to ld.bfd
using -fuse-ld=bfd with $(CC). This was taken from the arm vDSO
Makefile, as the comment notes, done in commit d2b30cd4b722 ("ARM:
8384/1: VDSO: force use of BFD linker").

Commit fe00e50b2db8 ("ARM: 8858/1: vdso: use $(LD) instead of $(CC) to
link VDSO") changed that Makefile to use $(LD) directly instead of
through $(CC), which matches how the rest of the kernel operates. Since
then, LD=ld.lld means that the arm vDSO will be linked with ld.lld,
which has shown no problems so far.

Allow ld.lld to link this vDSO as we do the regular arm vDSO. To do
this, we need to do a few things:

* Add a LD_COMPAT variable, which defaults to $(CROSS_COMPILE_COMPAT)ld
  with gcc and $(LD) if LLVM is 1, which will be ld.lld, or
  $(CROSS_COMPILE_COMPAT)ld if not, which matches the logic of the main
  Makefile. It is overrideable for further customization and avoiding
  breakage.

* Eliminate cc32-ldoption, which matches commit 055efab3120b ("kbuild:
  drop support for cc-ldoption").

With those, we can use $(LD_COMPAT) in cmd_ldvdso and change the flags
from compiler linker flags to linker flags directly. We eliminate
-mfloat-abi=soft because it is not handled by the linker.

Signed-off-by: Nathan Chancellor <natechancellor@gmail.com>
---
 arch/arm64/kernel/vdso32/Makefile | 23 ++++++++++++-----------
 1 file changed, 12 insertions(+), 11 deletions(-)

diff --git a/arch/arm64/kernel/vdso32/Makefile b/arch/arm64/kernel/vdso32/Makefile
index 3964738ebbde..0fb0a3a5e6a4 100644
--- a/arch/arm64/kernel/vdso32/Makefile
+++ b/arch/arm64/kernel/vdso32/Makefile
@@ -22,16 +22,21 @@ endif
 
 CC_COMPAT ?= $(CC)
 CC_COMPAT += $(CC_COMPAT_CLANG_FLAGS)
+
+ifeq ($(LLVM),1)
+LD_COMPAT ?= $(LD)
+else
+LD_COMPAT ?= $(CROSS_COMPILE_COMPAT)ld
+endif
 else
 CC_COMPAT ?= $(CROSS_COMPILE_COMPAT)gcc
+LD_COMPAT ?= $(CROSS_COMPILE_COMPAT)ld
 endif
 
 cc32-option = $(call try-run,\
         $(CC_COMPAT) $(1) -c -x c /dev/null -o "$$TMP",$(1),$(2))
 cc32-disable-warning = $(call try-run,\
 	$(CC_COMPAT) -W$(strip $(1)) -c -x c /dev/null -o "$$TMP",-Wno-$(strip $(1)))
-cc32-ldoption = $(call try-run,\
-        $(CC_COMPAT) $(1) -nostdlib -x c /dev/null -o "$$TMP",$(1),$(2))
 cc32-as-instr = $(call try-run,\
 	printf "%b\n" "$(1)" | $(CC_COMPAT) $(VDSO_AFLAGS) -c -x assembler -o "$$TMP" -,$(2),$(3))
 
@@ -114,14 +119,10 @@ dmbinstr := $(call cc32-as-instr,dmb ishld,-DCONFIG_AS_DMB_ISHLD=1)
 VDSO_CFLAGS += $(dmbinstr)
 VDSO_AFLAGS += $(dmbinstr)
 
-VDSO_LDFLAGS := $(VDSO_CPPFLAGS)
 # From arm vDSO Makefile
-VDSO_LDFLAGS += -Wl,-Bsymbolic -Wl,--no-undefined -Wl,-soname=linux-vdso.so.1
-VDSO_LDFLAGS += -Wl,-z,max-page-size=4096 -Wl,-z,common-page-size=4096
-VDSO_LDFLAGS += -nostdlib -shared -mfloat-abi=soft
-VDSO_LDFLAGS += -Wl,--hash-style=sysv
-VDSO_LDFLAGS += -Wl,--build-id
-VDSO_LDFLAGS += $(call cc32-ldoption,-fuse-ld=bfd)
+VDSO_LDFLAGS += -Bsymbolic --no-undefined -soname=linux-vdso.so.1
+VDSO_LDFLAGS += -z max-page-size=4096 -z common-page-size=4096
+VDSO_LDFLAGS += -nostdlib -shared --hash-style=sysv --build-id
 
 
 # Borrow vdsomunge.c from the arm vDSO
@@ -182,8 +183,8 @@ quiet_cmd_vdsold_and_vdso_check = LD32    $@
       cmd_vdsold_and_vdso_check = $(cmd_vdsold); $(cmd_vdso_check)
 
 quiet_cmd_vdsold = LD32    $@
-      cmd_vdsold = $(CC_COMPAT) -Wp,-MD,$(depfile) $(VDSO_LDFLAGS) \
-                   -Wl,-T $(filter %.lds,$^) $(filter %.o,$^) -o $@
+      cmd_vdsold = $(LD_COMPAT) $(VDSO_LDFLAGS) \
+                   -T $(filter %.lds,$^) $(filter %.o,$^) -o $@
 quiet_cmd_vdsocc = CC32    $@
       cmd_vdsocc = $(CC_COMPAT) -Wp,-MD,$(depfile) $(VDSO_CFLAGS) -c -o $@ $<
 quiet_cmd_vdsocc_gettimeofday = CC32    $@
-- 
2.27.0.rc0

[nathanchance]

Hmmm it seems that ld.lld breaks this somewhere because stress-ng's vDSO stressor no longer works (source appears to be here):

$ stress-ng --vdso 2
stress-ng: info:  [1046] vsdo stressor will be skipped, failed to find relevant vDSO functions
stress-ng: info:  [1046] defaulting to a 86400 second (1 day, 0.00 secs) run per stressor
stress-ng: error: [1046] No stress workers invoked (one or more were unsupported) 

As soon as I revert to ld.bfd via LD_COMPAT, everything works:

$ stress-ng --vdso 2
stress-ng: info:  [1102] defaulting to a 86400 second (1 day, 0.00 secs) run per stressor
stress-ng: info:  [1102] dispatching hogs: 2 vdso
stress-ng: info:  [1102] cache allocate: using defaults, can't determine cache details from sysfs
stress-ng: info:  [1103] stress-ng-vdso: exercising vDSO functions: __vdso_clock_gettime __vdso_gettimeofday

Dumping the sections, it looks like ld.bfd has a .got.plt section, whereas ld.lld has an .rodata section. Additionally, the _GLOBAL_OFFSET_TABLE_ and LINUX_2.6 are not present with ld.lld. Not sure if those differences would factor in.

$ readelf -S out/arm64-{bfd,lld}/arch/arm64/kernel/vdso32/vdso.so.raw

File: out/arm64-bfd/arch/arm64/kernel/vdso32/vdso.so.raw
There are 17 section headers, starting at offset 0xfcc:

Section Headers:
  [Nr] Name              Type            Addr     Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            00000000 000000 000000 00      0   0  0
  [ 1] .hash             HASH            000000b4 0000b4 00003c 04   A  2   0  4
  [ 2] .dynsym           DYNSYM          000000f0 0000f0 0000a0 10   A  3   1  4
  [ 3] .dynstr           STRTAB          00000190 000190 0000d5 00   A  0   0  1
  [ 4] .gnu.version      VERSYM          00000266 000266 000014 02   A  2   0  2
  [ 5] .gnu.version_d    VERDEF          0000027c 00027c 000038 00   A  3   2  4
  [ 6] .note             NOTE            000002b4 0002b4 000054 00   A  0   0  4
  [ 7] .dynamic          DYNAMIC         00000308 000308 000080 08  WA  3   0  4
  [ 8] .ARM.exidx        ARM_EXIDX       00000388 000388 000030 00  AL 10   0  4
  [ 9] .ARM.extab        PROGBITS        000003b8 0003b8 000030 00   A  0   0  4
  [10] .text             PROGBITS        000003e8 0003e8 00051e 00  AX  0   0  4
  [11] .got.plt          PROGBITS        00000908 000908 00000c 04  WA  0   0  4
  [12] .comment          PROGBITS        00000000 000914 000075 01  MS  0   0  1
  [13] .ARM.attributes   ARM_ATTRIBUTES  00000000 000989 00002c 00      0   0  1
  [14] .symtab           SYMTAB          00000000 0009b8 000340 10     15  43  4
  [15] .strtab           STRTAB          00000000 000cf8 000237 00      0   0  1
  [16] .shstrtab         STRTAB          00000000 000f2f 00009a 00      0   0  1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  y (purecode), p (processor specific)

File: out/arm64-lld/arch/arm64/kernel/vdso32/vdso.so.raw
There are 17 section headers, starting at offset 0xe98:

Section Headers:
  [Nr] Name              Type            Addr     Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            00000000 000000 000000 00      0   0  0
  [ 1] .hash             HASH            000000b4 0000b4 000050 04   A  2   0  4
  [ 2] .dynsym           DYNSYM          00000104 000104 000090 10   A  3   1  4
  [ 3] .dynstr           STRTAB          00000194 000194 0000d5 00   A  0   0  1
  [ 4] .gnu.version      VERSYM          0000026a 00026a 000012 02   A  2   0  2
  [ 5] .gnu.version_d    VERDEF          0000027c 00027c 000038 00   A  3   2  4
  [ 6] .ARM.exidx        ARM_EXIDX       000002b4 0002b4 000068 00  AL 11   0  4
  [ 7] .ARM.extab        PROGBITS        0000031c 00031c 000030 00   A  0   0  4
  [ 8] .note             NOTE            0000034c 00034c 000048 00   A  0   0  4
  [ 9] .dynamic          DYNAMIC         00000394 000394 000058 08  WA  3   0  4
  [10] .rodata           PROGBITS        000003ec 0003ec 000000 00  WA  0   0  1
  [11] .text             PROGBITS        000003ec 0003ec 00051e 00  AX  0   0  4
  [12] .comment          PROGBITS        00000000 00090a 0000d9 01  MS  0   0  1
  [13] .ARM.attributes   ARM_ATTRIBUTES  00000000 0009e3 00003c 00      0   0  1
  [14] .symtab           SYMTAB          00000000 000a20 0001f0 10     16  23  4
  [15] .shstrtab         STRTAB          00000000 000c10 000099 00      0   0  1
  [16] .strtab           STRTAB          00000000 000ca9 0001ef 00      0   0  1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  y (purecode), p (processor specific)

$ nm -S out/arm64-{bfd,lld}/arch/arm64/kernel/vdso32/vdso.so.raw

out/arm64-bfd/arch/arm64/kernel/vdso32/vdso.so.raw:
000003e8 t $a.0
000008dc t $a.0
00000638 t $a.2
000006a8 t $a.4
000002d8 r $d
00000388 r $d
000003b8 r $d
00000390 r $d
00000634 t $d.1
000006a4 t $d.3
000008d8 t $d.5
000008f4 t $t.1
000006a8 00000004 t __aeabi_unwind_cpp_pr0
000006ac 00000004 t __aeabi_unwind_cpp_pr1
000006b0 00000004 t __aeabi_unwind_cpp_pr2
000006b4 00000228 t __cvdso_clock_gettime_common
00000308 a _DYNAMIC
00000908 a _GLOBAL_OFFSET_TABLE_
000008ec 00000008 T __kernel_rt_sigreturn_arm
00000900 00000006 T __kernel_rt_sigreturn_thumb
000008e0 00000008 T __kernel_sigreturn_arm
000008f8 00000006 T __kernel_sigreturn_thumb
00000000 A LINUX_2.6
000002d8 00000018 r _note_14
000002f0 00000018 r _note_15
00000638 00000070 T __vdso_clock_getres
000003e8 00000050 T __vdso_clock_gettime
00000438 00000034 T __vdso_clock_gettime64
000008ec t VDSO_compat_rt_sigreturn_arm
00000900 t VDSO_compat_rt_sigreturn_thumb
000008e0 t VDSO_compat_sigreturn_arm
000008f8 t VDSO_compat_sigreturn_thumb
fffff000 r _vdso_data
0000046c 000001cc T __vdso_gettimeofday

out/arm64-lld/arch/arm64/kernel/vdso32/vdso.so.raw:
000003ec t $a.0
000008e0 t $a.0
0000063c t $a.2
000006ac t $a.4
00000638 t $d.1
000006a8 t $d.3
000008dc t $d.5
000008f8 t $t.1
000006ac 00000004 t __aeabi_unwind_cpp_pr0
000006b0 00000004 t __aeabi_unwind_cpp_pr1
000006b4 00000004 t __aeabi_unwind_cpp_pr2
000006b8 00000228 t __cvdso_clock_gettime_common
00000394 d _DYNAMIC
000008f0 00000008 T __kernel_rt_sigreturn_arm
00000904 00000006 T __kernel_rt_sigreturn_thumb
000008e4 00000008 T __kernel_sigreturn_arm
000008fc 00000006 T __kernel_sigreturn_thumb
0000034c 00000018 r _note_14
00000364 00000018 r _note_15
0000063c 00000070 T __vdso_clock_getres
000003ec 00000050 T __vdso_clock_gettime
0000043c 00000034 T __vdso_clock_gettime64
000008f0 t VDSO_compat_rt_sigreturn_arm
00000904 t VDSO_compat_rt_sigreturn_thumb
000008e4 t VDSO_compat_sigreturn_arm
000008fc t VDSO_compat_sigreturn_thumb
fffff000 r _vdso_data
00000470 000001cc T __vdso_gettimeofday

The linker script is here: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm64/kernel/vdso32/vdso.lds.S

The full Makefile is here: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm64/kernel/vdso32/Makefile

cc @MaskRay for further input.

[nathanchance]

I ended up testing an ARCH=arm kernel on my Raspberry Pi and it turns out the vDSO does not work when linked with ld.lld there either, failing in the same manner as the arm64 vdso32.

[nickdesaulniers]

This is blocking moving android to LLVM=1. Raising priority.

[nickdesaulniers]

Let's compare the symbols with llvm-readelf -s rather than nm. (nm should be avoided in preference to llvm-readelf -s forever). That should tell us more about the symbol types.

[nathanchance]

Sure. After reapplying my patch on mainline:

$ make -skj"$(nproc)" ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- CROSS_COMPILE_COMPAT=arm-linux-gnueabi- LLVM=1 O=out/arm64-ld.lld distclean defconfig arch/arm64/kernel/vdso32/

$ make -skj"$(nproc)" ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- CROSS_COMPILE_COMPAT=arm-linux-gnueabi- LD_COMPAT=arm-linux-gnueabi-ld LLVM=1 O=out/arm64-ld.bfd distclean defconfig arch/arm64/kernel/vdso32/

$ llvm-readelf -s out/arm64-ld.{bfd,lld}/arch/arm64/kernel/vdso32/vdso.so.raw

File: out/arm64-ld.bfd/arch/arm64/kernel/vdso32/vdso.so.raw

Symbol table '.dynsym' contains 6 entries:
   Num:    Value  Size Type    Bind   Vis       Ndx Name
     0: 00000000     0 NOTYPE  LOCAL  DEFAULT   UND 
     1: 00000000     0 OBJECT  GLOBAL DEFAULT   ABS LINUX_2.6@@LINUX_2.6
     2: 00000955   552 FUNC    GLOBAL DEFAULT     9 __vdso_gettimeofday@@LINUX_2.6
     3: 00000b7d    96 FUNC    GLOBAL DEFAULT     9 __vdso_clock_getres@@LINUX_2.6
     4: 000005ed   872 FUNC    GLOBAL DEFAULT     9 __vdso_clock_gettime64@@LINUX_2.6
     5: 000002d1   796 FUNC    GLOBAL DEFAULT     9 __vdso_clock_gettime@@LINUX_2.6

Symbol table '.symtab' contains 48 entries:
   Num:    Value  Size Type    Bind   Vis       Ndx Name
     0: 00000000     0 NOTYPE  LOCAL  DEFAULT   UND 
     1: 000000b4     0 SECTION LOCAL  DEFAULT     1 .hash
     2: 000000e0     0 SECTION LOCAL  DEFAULT     2 .dynsym
     3: 00000140     0 SECTION LOCAL  DEFAULT     3 .dynstr
     4: 000001b0     0 SECTION LOCAL  DEFAULT     4 .gnu.version
     5: 000001bc     0 SECTION LOCAL  DEFAULT     5 .gnu.version_d
     6: 000001f4     0 SECTION LOCAL  DEFAULT     6 .note
     7: 00000248     0 SECTION LOCAL  DEFAULT     7 .dynamic
     8: 000002c8     0 SECTION LOCAL  DEFAULT     8 .ARM.exidx
     9: 000002d0     0 SECTION LOCAL  DEFAULT     9 .text
    10: 00000be4     0 SECTION LOCAL  DEFAULT    10 .got.plt
    11: 00000000     0 SECTION LOCAL  DEFAULT    11 .debug_abbrev
    12: 00000000     0 SECTION LOCAL  DEFAULT    12 .debug_info
    13: 00000000     0 SECTION LOCAL  DEFAULT    13 .debug_str
    14: 00000000     0 SECTION LOCAL  DEFAULT    14 .comment
    15: 00000000     0 SECTION LOCAL  DEFAULT    15 .ARM.attributes
    16: 00000000     0 SECTION LOCAL  DEFAULT    16 .debug_line
    17: 00000000     0 SECTION LOCAL  DEFAULT    17 .debug_loc
    18: 00000000     0 SECTION LOCAL  DEFAULT    18 .debug_ranges
    19: 00000000     0 SECTION LOCAL  DEFAULT    19 .debug_frame
    20: 00000000     0 FILE    LOCAL  DEFAULT   ABS note.c
    21: 00000218    24 OBJECT  LOCAL  DEFAULT     6 _note_14
    22: 00000218     0 NOTYPE  LOCAL  DEFAULT     6 $d
    23: 00000230    24 OBJECT  LOCAL  DEFAULT     6 _note_15
    24: 00000000     0 FILE    LOCAL  DEFAULT   ABS vgettimeofday.c
    25: 000002d0     0 NOTYPE  LOCAL  DEFAULT     9 $t
    26: 000005e0     0 NOTYPE  LOCAL  DEFAULT     9 $d
    27: 000002c8     0 NOTYPE  LOCAL  DEFAULT     8 $d
    28: 000005ec     0 NOTYPE  LOCAL  DEFAULT     9 $t
    29: 00000948     0 NOTYPE  LOCAL  DEFAULT     9 $d
    30: 00000954     0 NOTYPE  LOCAL  DEFAULT     9 $t
    31: 00000b70     0 NOTYPE  LOCAL  DEFAULT     9 $d
    32: 00000b7c     0 NOTYPE  LOCAL  DEFAULT     9 $t
    33: 00000bd4     0 NOTYPE  LOCAL  DEFAULT     9 $d
    34: 00000bdc     0 NOTYPE  LOCAL  DEFAULT     9 $t
    35: 00000000     0 FILE    LOCAL  DEFAULT   ABS 
    36: fffff000     0 NOTYPE  LOCAL  DEFAULT   ABS _timens_data
    37: 00000248     0 OBJECT  LOCAL  DEFAULT   ABS _DYNAMIC
    38: 00000bdd     2 FUNC    LOCAL  DEFAULT     9 __aeabi_unwind_cpp_pr0
    39: 00000be1     2 FUNC    LOCAL  DEFAULT     9 __aeabi_unwind_cpp_pr2
    40: ffffe000     0 NOTYPE  LOCAL  DEFAULT     1 _vdso_data
    41: 00000be4     0 OBJECT  LOCAL  DEFAULT   ABS _GLOBAL_OFFSET_TABLE_
    42: 00000bdf     2 FUNC    LOCAL  DEFAULT     9 __aeabi_unwind_cpp_pr1
    43: 00000000     0 OBJECT  GLOBAL DEFAULT   ABS LINUX_2.6
    44: 00000955   552 FUNC    GLOBAL DEFAULT     9 __vdso_gettimeofday
    45: 00000b7d    96 FUNC    GLOBAL DEFAULT     9 __vdso_clock_getres
    46: 000005ed   872 FUNC    GLOBAL DEFAULT     9 __vdso_clock_gettime64
    47: 000002d1   796 FUNC    GLOBAL DEFAULT     9 __vdso_clock_gettime

File: out/arm64-ld.lld/arch/arm64/kernel/vdso32/vdso.so.raw

Symbol table '.dynsym' contains 5 entries:
   Num:    Value  Size Type    Bind   Vis       Ndx Name
     0: 00000000     0 NOTYPE  LOCAL  DEFAULT   UND 
     1: 000002c9   796 FUNC    GLOBAL DEFAULT    11 __vdso_clock_gettime@@LINUX_2.6
     2: 000005e5   872 FUNC    GLOBAL DEFAULT    11 __vdso_clock_gettime64@@LINUX_2.6
     3: 0000094d   552 FUNC    GLOBAL DEFAULT    11 __vdso_gettimeofday@@LINUX_2.6
     4: 00000b75    96 FUNC    GLOBAL DEFAULT    11 __vdso_clock_getres@@LINUX_2.6

Symbol table '.symtab' contains 25 entries:
   Num:    Value  Size Type    Bind   Vis       Ndx Name
     0: 00000000     0 NOTYPE  LOCAL  DEFAULT   UND 
     1: 00000000     0 FILE    LOCAL  DEFAULT   ABS note.c
     2: 00000228    24 OBJECT  LOCAL  DEFAULT     8 _note_14
     3: 00000228     0 NOTYPE  LOCAL  DEFAULT     8 $d
     4: 00000240    24 OBJECT  LOCAL  DEFAULT     8 _note_15
     5: 00000000     0 FILE    LOCAL  DEFAULT   ABS vgettimeofday.c
     6: 000002c8     0 NOTYPE  LOCAL  DEFAULT    11 $t
     7: 000005d8     0 NOTYPE  LOCAL  DEFAULT    11 $d
     8: 000005e4     0 NOTYPE  LOCAL  DEFAULT    11 $t
     9: 00000940     0 NOTYPE  LOCAL  DEFAULT    11 $d
    10: 0000094c     0 NOTYPE  LOCAL  DEFAULT    11 $t
    11: 00000b68     0 NOTYPE  LOCAL  DEFAULT    11 $d
    12: 00000b74     0 NOTYPE  LOCAL  DEFAULT    11 $t
    13: 00000bcc     0 NOTYPE  LOCAL  DEFAULT    11 $d
    14: 00000bd4     0 NOTYPE  LOCAL  DEFAULT    11 $t
    15: 00000bd5     2 FUNC    LOCAL  DEFAULT    11 __aeabi_unwind_cpp_pr0
    16: 00000bd7     2 FUNC    LOCAL  DEFAULT    11 __aeabi_unwind_cpp_pr1
    17: 00000bd9     2 FUNC    LOCAL  DEFAULT    11 __aeabi_unwind_cpp_pr2
    18: ffffe000     0 NOTYPE  LOCAL  HIDDEN      1 _vdso_data
    19: fffff000     0 NOTYPE  LOCAL  HIDDEN      1 _timens_data
    20: 00000270     0 NOTYPE  LOCAL  HIDDEN      9 _DYNAMIC
    21: 000002c9   796 FUNC    GLOBAL DEFAULT    11 __vdso_clock_gettime
    22: 000005e5   872 FUNC    GLOBAL DEFAULT    11 __vdso_clock_gettime64
    23: 0000094d   552 FUNC    GLOBAL DEFAULT    11 __vdso_gettimeofday
    24: 00000b75    96 FUNC    GLOBAL DEFAULT    11 __vdso_clock_getres

[nickdesaulniers]

I don't see any differences/missing function type symbols. LLD is missing a couple section symbols, but they have size 0 with BFD. Not sure what LINUX_2.6@@LINUX_2.6 is. There's quite a few non-zero-sized section symbols missing with LLD. Missing _GLOBAL_OFFSET_TABLE_ and .got.plt is immediately concerning. But those are just symbols; not indicative of the section itself being missing.

stress-ng: info: [1103] stress-ng-vdso: exercising vDSO functions: __vdso_clock_gettime __vdso_gettimeofday

https://fossies.org/linux/stress-ng/stress-vdso.c

Looks like stress-ng calls vdso_sym_list_str printing the error on NULL returned. vdso_sym_list_str iterates vdso_sym_list, which was built via dl_wrapback().

stress-ng: info: [1046] vsdo stressor will be skipped, failed to find relevant vDSO functions

means dl_wrapback() did not modify vdso_sym_list . From here, I'm not really sure if there's a bug in dl_wrapback or not. I'd be curious what would happen if the first iteration of the while loop had a DT_SYMTAB case, then hash and strtab would be NULL and we'd skip the symbol (or is that somehow impossible?). Probably need to build stress-ng from source with debug info and break on dl_wrapback() to see where behavior diverges. cc @ColinIanKing

Wait, are there no symbol hashes? @nathanchance can you redump with llvm-readelf -S (capital S for sections)? This isn't a missing -Wl,--hash-style=sysv linker flag is it? (Let's see the sections to find out). (Should default to --hash-style=both...hmmm. (No, it's in @nathanchance 's diff)

@MaskRay does anything look obviously wrong to you in stress-ng's dl_wrapback() symbol table parsing, or shall we fire up a debugger?

[nathanchance]

I think I had this earlier but here it is again!

$ llvm-readelf -S out/arm64-ld.{bfd,lld}/arch/arm64/kernel/vdso32/vdso.so.raw

File: out/arm64-ld.bfd/arch/arm64/kernel/vdso32/vdso.so.raw
There are 23 section headers, starting at offset 0x548c:

Section Headers:
  [Nr] Name              Type            Address  Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            00000000 000000 000000 00      0   0  0
  [ 1] .hash             HASH            000000b4 0000b4 00002c 04   A  2   0  4
  [ 2] .dynsym           DYNSYM          000000e0 0000e0 000060 10   A  3   1  4
  [ 3] .dynstr           STRTAB          00000140 000140 00006f 00   A  0   0  1
  [ 4] .gnu.version      VERSYM          000001b0 0001b0 00000c 02   A  2   0  2
  [ 5] .gnu.version_d    VERDEF          000001bc 0001bc 000038 00   A  3   2  4
  [ 6] .note             NOTE            000001f4 0001f4 000054 00   A  0   0  4
  [ 7] .dynamic          DYNAMIC         00000248 000248 000080 08  WA  3   0  4
  [ 8] .ARM.exidx        ARM_EXIDX       000002c8 0002c8 000008 00  AL  9   0  4
  [ 9] .text             PROGBITS        000002d0 0002d0 000914 00  AX  0   0  4
  [10] .got.plt          PROGBITS        00000be4 000be4 00000c 04  WA  0   0  4
  [11] .debug_abbrev     PROGBITS        00000000 000bf0 0002e1 00      0   0  1
  [12] .debug_info       PROGBITS        00000000 000ed1 00184f 00      0   0  1
  [13] .debug_str        PROGBITS        00000000 002720 0006dd 01  MS  0   0  1
  [14] .comment          PROGBITS        00000000 002dfd 000075 01  MS  0   0  1
  [15] .ARM.attributes   ARM_ATTRIBUTES  00000000 002e72 00003d 00      0   0  1
  [16] .debug_line       PROGBITS        00000000 002eaf 000ac8 00      0   0  1
  [17] .debug_loc        PROGBITS        00000000 003977 00114d 00      0   0  1
  [18] .debug_ranges     PROGBITS        00000000 004ac4 000408 00      0   0  1
  [19] .debug_frame      PROGBITS        00000000 004ecc 0000d0 00      0   0  4
  [20] .symtab           SYMTAB          00000000 004f9c 000300 10     21  43  4
  [21] .strtab           STRTAB          00000000 00529c 00010a 00      0   0  1
  [22] .shstrtab         STRTAB          00000000 0053a6 0000e6 00      0   0  1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  y (purecode), p (processor specific)

File: out/arm64-ld.lld/arch/arm64/kernel/vdso32/vdso.so.raw
There are 24 section headers, starting at offset 0x53d8:

Section Headers:
  [Nr] Name              Type            Address  Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            00000000 000000 000000 00      0   0  0
  [ 1] .hash             HASH            000000b4 0000b4 000030 04   A  2   0  4
  [ 2] .dynsym           DYNSYM          000000e4 0000e4 000050 10   A  3   1  4
  [ 3] .dynstr           STRTAB          00000134 000134 00006f 00   A  0   0  1
  [ 4] .gnu.version      VERSYM          000001a4 0001a4 00000a 02   A  2   0  2
  [ 5] .gnu.version_d    VERDEF          000001b0 0001b0 000038 00   A  3   2  4
  [ 6] .ARM.exidx        ARM_EXIDX       000001e8 0001e8 000040 00  AL 11   0  4
  [ 7] .ARM.extab        PROGBITS        00000228 000228 000000 00   A  0   0  1
  [ 8] .note             NOTE            00000228 000228 000048 00   A  0   0  4
  [ 9] .dynamic          DYNAMIC         00000270 000270 000058 08  WA  3   0  4
  [10] .rodata           PROGBITS        000002c8 0002c8 000000 00  WA  0   0  1
  [11] .text             PROGBITS        000002c8 0002c8 000914 00  AX  0   0  4
  [12] .debug_abbrev     PROGBITS        00000000 000bdc 0002e1 00      0   0  1
  [13] .debug_info       PROGBITS        00000000 000ebd 00184f 00      0   0  1
  [14] .debug_str        PROGBITS        00000000 00270c 00075a 01  MS  0   0  1
  [15] .comment          PROGBITS        00000000 002e66 0000d9 01  MS  0   0  1
  [16] .debug_line       PROGBITS        00000000 002f3f 000ac8 00      0   0  1
  [17] .ARM.attributes   ARM_ATTRIBUTES  00000000 003a07 00003d 00      0   0  1
  [18] .debug_loc        PROGBITS        00000000 003a44 00114d 00      0   0  1
  [19] .debug_ranges     PROGBITS        00000000 004b91 000408 00      0   0  1
  [20] .debug_frame      PROGBITS        00000000 004f9c 0000d0 00      0   0  4
  [21] .symtab           SYMTAB          00000000 00506c 000190 10     23  21  4
  [22] .shstrtab         STRTAB          00000000 0051fc 0000f0 00      0   0  1
  [23] .strtab           STRTAB          00000000 0052ec 0000ea 00      0   0  1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  y (purecode), p (processor specific)

Unfortunately, I will not be online on Friday October 2nd. I can provide any follow up information or test things out over the weekend or Monday at the latest.

[nathanchance]

So I did some more testing in a virtualized Debian environment with stress-ng and it seems like the regular arm64 and x86_64 vDSOs are broken with ld.lld...

With just LLVM=1:

$ make -skj"$(nproc)" LLVM=1 distclean defconfig bzImage
...

$ ../../cbl/github/boot-qemu.sh -a x86_64 -k . -i --debian
...
[    0.000000] Linux version 5.9.0-rc7-next-20201002 (nathan@ubuntu-m3-large-x86) (ClangBuiltLinux clang version 12.0.0 (https://github.com/llvm/llvm-project 6e25586990b93e2c9eaaa4f473b6720ccd646c46), LLD 12.0.0 (https://github.com/llvm/llvm-project 6e25586990b93e2c9eaaa4f473b6720ccd646c46)) #1 SMP Tue Oct 6 01:20:09 MST 2020
...
root@ubuntu-m3-large-x86:~# stress-ng --vdso 2
stress-ng: info:  [575] vsdo stressor will be skipped, failed to find relevant vDSO functions
stress-ng: info:  [575] Working directory /root is not read/writeable, some I/O tests may fail
stress-ng: info:  [575] defaulting to a 86400 second (1 day, 0.00 secs) run per stressor
stress-ng: error: [575] No stress workers invoked (one or more were unsupported)
...

With LD=ld LLVM=1:

$ make -skj"$(nproc)" LD=ld LLVM=1 distclean defconfig bzImage
...

$ ../../cbl/github/boot-qemu.sh -a x86_64 -k . -i --debian
...
[    0.000000] Linux version 5.9.0-rc7-next-20201002 (nathan@ubuntu-m3-large-x86) (ClangBuiltLinux clang version 12.0.0 (https://github.com/llvm/llvm-project 6e25586990b93e2c9eaaa4f473b6720ccd646c46), GNU ld (GNU Binutils) 2.35.50.20201006) #1 SMP Tue Oct 6 01:24:13 MST 2020
...
root@ubuntu-m3-large-x86:~# stress-ng --vdso 2
stress-ng: info:  [577] Working directory /root is not read/writeable, some I/O tests may fail
stress-ng: info:  [577] defaulting to a 86400 second (1 day, 0.00 secs) run per stressor
stress-ng: info:  [577] dispatching hogs: 2 vdso
stress-ng: info:  [578] stress-ng-vdso: exercising vDSO functions: clock_gettime time gettimeofday getcpu
...

I can provide more information about the Debian testing, it currently sitting in a branch here.

[nickdesaulniers]

patch needs to be rebased due to conflict with a968433 on -next. (and https://lore.kernel.org/linux-next/20201012141032.6333-1-cai@redhat.com/ to work around an unrelated issue).

[nathanchance]

I can rebase it against -next but I think qualification should happen against mainline first for noise sake.

[nickdesaulniers]

Our presubmit runs kselftests which contain vdso tests for x86 and arm, both 32b and 64b variants. All presubmits are now passing.

mainline: https://android-review.googlesource.com/c/kernel/common/+/1457538
5.4: https://android-review.googlesource.com/c/kernel/common/+/1457540
4.19: https://android-review.googlesource.com/c/kernel/common/+/1457543

Please send this upstream and I will update the (am from XXX) to include the correct lore.kernel.org link.

[nathanchance]

Patch submitted: https://lore.kernel.org/r/20201013033947.2257501-1-natechancellor@gmail.com

[nathanchance]

v2: https://lore.kernel.org/lkml/20201020011406.1818918-1-natechancellor@gmail.com/

[nathanchance]

Accepted: https://git.kernel.org/arm64/c/0fa97e9403c7

[nathanchance]

Merged into mainline: https://git.kernel.org/torvalds/c/0fa97e9403c7