lld uses blx instruction, no object with architecture supporting feature detected

[nickdesaulniers]

0day bot reported this arm randconfig: https://groups.google.com/g/clang-built-linux/c/n7DXLyrlo-0/m/GTTaxotDBwAJ

>> ld.lld: warning: lld uses blx instruction, no object with architecture supporting feature detected

[nathanchance]

$ curl -LSs 'https://lore.kernel.org/lkml/202004010505.30nygaXZ%25lkp@intel.com/2-a.bin' | gzip -d > out/arm32/.config

$ make -j$(nproc) -s ARCH=arm CC=clang CROSS_COMPILE=arm-linux-gnueabi- LD=ld.lld O=out/arm32 olddefconfig all
/home/nathan/src/linux/arch/arm/mach-sa1100/generic.c:380:24: warning: unused variable 'standard_io_desc' [-Wunused-variable]
static struct map_desc standard_io_desc[] __initdata = {
                       ^
1 warning generated.
/home/nathan/src/linux/arch/arm/mm/copypage-v4mc.c:70:3: error: implicit declaration of function '__flush_dcache_page' [-Werror,-Wimplicit-function-declaration]
                __flush_dcache_page(page_mapping_file(from), from);
                ^
/home/nathan/src/linux/arch/arm/mm/copypage-v4mc.c:70:3: note: did you mean 'flush_dcache_page'?
/home/nathan/src/linux/arch/arm/include/asm/cacheflush.h:292:13: note: 'flush_dcache_page' declared here
extern void flush_dcache_page(struct page *);
            ^
/home/nathan/src/linux/arch/arm/mm/copypage-v4mc.c:74:2: error: implicit declaration of function 'set_top_pte' [-Werror,-Wimplicit-function-declaration]
        set_top_pte(COPYPAGE_MINICACHE, mk_pte(from, minicache_pgprot));
        ^
/home/nathan/src/linux/arch/arm/mm/copypage-v4mc.c:74:34: error: implicit declaration of function 'mk_pte' [-Werror,-Wimplicit-function-declaration]
        set_top_pte(COPYPAGE_MINICACHE, mk_pte(from, minicache_pgprot));
                                        ^
/home/nathan/src/linux/arch/arm/mm/copypage-v4mc.c:74:47: error: use of undeclared identifier 'L_PTE_PRESENT'
        set_top_pte(COPYPAGE_MINICACHE, mk_pte(from, minicache_pgprot));
                                                     ^
/home/nathan/src/linux/arch/arm/mm/copypage-v4mc.c:23:35: note: expanded from macro 'minicache_pgprot'
#define minicache_pgprot __pgprot(L_PTE_PRESENT | L_PTE_YOUNG | \
                                  ^
/home/nathan/src/linux/arch/arm/mm/copypage-v4mc.c:74:47: error: use of undeclared identifier 'L_PTE_YOUNG'
/home/nathan/src/linux/arch/arm/mm/copypage-v4mc.c:23:51: note: expanded from macro 'minicache_pgprot'
#define minicache_pgprot __pgprot(L_PTE_PRESENT | L_PTE_YOUNG | \
                                                  ^
/home/nathan/src/linux/arch/arm/mm/copypage-v4mc.c:74:47: error: use of undeclared identifier 'L_PTE_MT_MINICACHE'
/home/nathan/src/linux/arch/arm/mm/copypage-v4mc.c:24:7: note: expanded from macro 'minicache_pgprot'
                                  L_PTE_MT_MINICACHE)
                                  ^
/home/nathan/src/linux/arch/arm/mm/copypage-v4mc.c:74:14: error: use of undeclared identifier 'COPYPAGE_MINICACHE'
        set_top_pte(COPYPAGE_MINICACHE, mk_pte(from, minicache_pgprot));
                    ^
/home/nathan/src/linux/arch/arm/mm/copypage-v4mc.c:76:28: error: use of undeclared identifier 'COPYPAGE_MINICACHE'
        mc_copy_user_page((void *)COPYPAGE_MINICACHE, kto);
                                  ^
/home/nathan/src/linux/arch/arm/mm/copypage-v4mc.c:109:21: error: variable has incomplete type 'struct cpu_user_fns'
struct cpu_user_fns v4_mc_user_fns __initdata = {
                    ^
/home/nathan/src/linux/arch/arm/mm/copypage-v4mc.c:109:8: note: forward declaration of 'struct cpu_user_fns'
struct cpu_user_fns v4_mc_user_fns __initdata = {
       ^
9 errors generated.
make[3]: *** [/home/nathan/src/linux/scripts/Makefile.build:267: arch/arm/mm/copypage-v4mc.o] Error 1
make[2]: *** [/home/nathan/src/linux/Makefile:1702: arch/arm/mm] Error 2
make[2]: *** Waiting for unfinished jobs....
/home/nathan/src/linux/arch/arm/mach-sa1100/assabet.c:631:38: error: use of undeclared identifier 'PMD_MASK'
        unsigned long phys = __PREG(GPLR) & PMD_MASK;
                                            ^
/home/nathan/src/linux/arch/arm/mach-sa1100/assabet.c:633:13: error: use of undeclared identifier 'pmdval_t'
        int prot = PMD_TYPE_SECT | PMD_SECT_AP_WRITE | PMD_DOMAIN(DOMAIN_IO);
                   ^
/home/nathan/src/linux/arch/arm/include/asm/pgtable-2level-hwdef.h:19:29: note: expanded from macro 'PMD_TYPE_SECT'
#define PMD_TYPE_SECT           (_AT(pmdval_t, 2) << 0)
                                     ^
/home/nathan/src/linux/arch/arm/mach-sa1100/assabet.c:633:29: error: use of undeclared identifier 'pmdval_t'
        int prot = PMD_TYPE_SECT | PMD_SECT_AP_WRITE | PMD_DOMAIN(DOMAIN_IO);
                                   ^
/home/nathan/src/linux/arch/arm/include/asm/pgtable-2level-hwdef.h:32:32: note: expanded from macro 'PMD_SECT_AP_WRITE'
#define PMD_SECT_AP_WRITE       (_AT(pmdval_t, 1) << 10)
                                     ^
/home/nathan/src/linux/arch/arm/mach-sa1100/assabet.c:633:49: error: use of undeclared identifier 'pmdval_t'
        int prot = PMD_TYPE_SECT | PMD_SECT_AP_WRITE | PMD_DOMAIN(DOMAIN_IO);
                                                       ^
/home/nathan/src/linux/arch/arm/include/asm/pgtable-2level-hwdef.h:22:29: note: expanded from macro 'PMD_DOMAIN'
#define PMD_DOMAIN(x)           (_AT(pmdval_t, (x)) << 5)
                                     ^
/home/nathan/src/linux/arch/arm/mach-sa1100/assabet.c:638:2: error: implicit declaration of function 'flush_pmd_entry' [-Werror,-Wimplicit-function-declaration]
        flush_pmd_entry(pmd);
        ^
5 errors generated.
make[3]: *** [/home/nathan/src/linux/scripts/Makefile.build:267: arch/arm/mach-sa1100/assabet.o] Error 1
make[2]: *** [/home/nathan/src/linux/Makefile:1702: arch/arm/mach-sa1100] Error 2
ld.lld: warning: lld uses blx instruction, no object with architecture supporting feature detected
ld.lld: warning: lld uses blx instruction, no object with architecture supporting feature detected
ld.lld: warning: lld uses blx instruction, no object with architecture supporting feature detected
ld.lld: warning: lld uses blx instruction, no object with architecture supporting feature detected
ld.lld: warning: lld uses blx instruction, no object with architecture supporting feature detected
ld.lld: warning: lld uses blx instruction, no object with architecture supporting feature detected
ld.lld: warning: lld uses blx instruction, no object with architecture supporting feature detected
ld.lld: warning: lld uses blx instruction, no object with architecture supporting feature detected
ld.lld: warning: lld uses blx instruction, no object with architecture supporting feature detected
ld.lld: warning: lld uses blx instruction, no object with architecture supporting feature detected
ld.lld: warning: lld uses blx instruction, no object with architecture supporting feature detected
ld.lld: warning: lld uses blx instruction, no object with architecture supporting feature detected
ld.lld: warning: lld uses blx instruction, no object with architecture supporting feature detected
ld.lld: warning: lld uses blx instruction, no object with architecture supporting feature detected
ld.lld: warning: lld uses blx instruction, no object with architecture supporting feature detected
ld.lld: warning: lld uses blx instruction, no object with architecture supporting feature detected
ld.lld: warning: lld uses blx instruction, no object with architecture supporting feature detected
make[1]: *** [/home/nathan/src/linux/Makefile:328: __build_one_by_one] Error 2
make: *** [Makefile:180: sub-make] Error 2

https://github.com/llvm/llvm-project/blob/f203100ebe22bf97a4268a562cdbef22d14db915/lld/ELF/Driver.cpp#L1996-L2003

https://github.com/llvm/llvm-project/blob/f203100ebe22bf97a4268a562cdbef22d14db915/lld/test/ELF/arm-blx-v4t.s

$ rg CPU_32 out/arm32/.config
259:CONFIG_CPU_32v4=y

Seems expected given the config. If we had a config for ld.lld, we could just avoid selecting CPU__32v4 or CPU_32v4T if using ld.lld.

[nickdesaulniers]

yes, maybe a way forward is to upstream Android's CONFIG_LD_IS_ patches (cc @samitolvanen ) then we can disable certain configs when it's selected. cc @masahir0y for thoughts on that.

We could also just mark CONFIG_CPU_32v4 disabled for Clang. I think your report above is the furthest anyone's testing Linux ARMv4 w/ Clang.

[masahir0y]

I am fine with LD_IS_LLD if it is useful.

[MaskRay]

LD_IS_LLD looks fine with me. The upstream tends to not support rarely used/obsoleted/error-prone features/ABI variants. Some CONFIG_* may never make sense for lld.

I wonder whether this can also benefit big-endian powerpc64 #602. There is no interest to support elfv1. If we enable elfv2 ("Power Architecture 64-Bit ELF V2 ABI Specification") via some config, then this config should always to y if linked with lld.

[nathanchance]

Hmmm, looking over this, fixing this up is going to be rather invasive. Reason being is that CONFIG_CPU_32v4{,T} are not user selectable, they are only selected by configs that need them. As a result, we can't make CONFIG_CPU_32v4{,T} depend on CONFIG_LD_IS_LLD otherwise we'll get some Kconfig warnings because select overrides depends on.

WARNING: unmet direct dependencies detected for CPU_32v4
  Depends on [n]: !LD_IS_LLD [=y]
  Selected by [y]:
  - CPU_SA1100 [=y]

We could just never select CONFIG_CPU_32v4{,T} when using CONFIG_LD_IS_LLD but that just blows up the config in another way because __LINUX_ARM_ARCH__ is unset and evaluates to zero.

The real solution it seems is to avoid selecting the machines that select these configs in arch/arm/Kconfig but it gets reallllly messy from there.

[nickdesaulniers]

updateSupportedARMFeatures in llvm-project/lld/ELF/InputFiles.cpp has a comment:

    // Architectures prior to v5 do not support BLX instruction

but there's also a FIXME in llvm-project/lld/ELF/Driver.cpp:

    // FIXME: These warnings can be removed when lld only uses these features
    // when the input objects have been compiled with an architecture that
    // supports them.

Indeed, it looks like a Kconfig fix is going to be quite invasive to disable LLD pre-ARMv5.

[nickdesaulniers]

because select overrides depends on

That's kind of frustrating. That means then that we have to add depends on !LD_IS_LLD to many leaf configs, rather than root configs.

[nickdesaulniers]

Easily reproducible with shannon_defconfig.

llvm commit 57eb0469847b ("[ELF] Read ARM BuildAttributes section to determine supported features.") has more context.

To silence any warning for shannon_defconfig:

diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
index e6e08d8a45fc..3379ae750a84 100644
--- a/arch/arm/Kconfig
+++ b/arch/arm/Kconfig
@@ -459,6 +459,7 @@ config ARCH_RPC
 
 config ARCH_SA1100
        bool "SA1100-based"
+       depends on !LD_IS_LLD
        select ARCH_MTD_XIP
        select ARCH_SPARSEMEM_ENABLE
        select CLKSRC_MMIO

It builds but it's obviously going to not boot/work properly, since that disables CPU_SA1100 and CPU_32v4. Perhaps that's an improvement for randconfig testing though?

[nickdesaulniers]

reported upstream: https://llvm.org/pr51422

[nickdesaulniers]

I wonder if https://reviews.llvm.org/rG62c605771a30 is relevant?

[nickdesaulniers]

llvm/llvm-project#50764 was just closed out. We should see if we can update intel-lab-lkp@6a7ee50 for lld-16.

cc @arndb @stuij

[nathanchance]

Patch submitted for the kernel: https://lore.kernel.org/20230118-v4-v4t-lld-16-v1-1-e3d9a00ae47c@kernel.org/

[nathanchance]

Patch submitted to Russell's patch tracker: https://www.armlinux.org.uk/developer/patches/viewpatch.php?id=9289/1

[nathanchance]

Patch accepted: http://git.armlinux.org.uk/cgit/linux-arm.git/commit/?id=5eb6e280432ddc9b755193552f3a070da8d7455c

[nathanchance]

Patch is now available in mainline: https://git.kernel.org/linus/5eb6e280432ddc9b755193552f3a070da8d7455c