Status: Building with Clang's integrated assembler aka LLVM_IAS=1 #1049

dileks · 2020-06-11T09:40:35Z

Hi,

what combination of recent Linux-kernel and recent llvm-toolchain is safe to build with enabled Clang's (I)ntegrated (AS)sembler aka set make-option LLVM_IAS=1?

kernel-doc llvm.rst [1] says:

Currently, the integrated assembler is disabled by default. You can pass
LLVM_IAS=1 to enable it.

I am here on Debian/testing AMD64 with Linux v5.7.2 and llvm-toolchain-10 (release/10.x Git branch):

$ clang-10 --version
clang version 10.0.1 (https://github.com/llvm/llvm-project b6efa2365812f31667485c8948d49621ebf952f2)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/dileks/src/llvm-toolchain/install/bin

I have cherry-picked commit 51da9df from upstream:

$ git branch --show-current
for-5.7/elfnote-llvm-ias-ndesaulniers

$ git log --oneline v5.7.2..
604f98ca53d6 (HEAD -> for-5.7/elfnote-llvm-ias-ndesaulniers) elfnote: mark all .note sections SHF_ALLOC

Is this enough to have good chances for a successful build?

Thanks in advance.

Regards,

Sedat -

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/kbuild/llvm.rst#n62

The text was updated successfully, but these errors were encountered:

dileks · 2020-06-11T10:13:50Z

FYI:
I have added LLVM_IAS=1 to my make-options and cherry-picked elfnote: mark all .note sections SHF_ALLOC.
Let's see...

dileks · 2020-06-11T10:31:34Z

Short intermezzo:

  clang-10 -Wp,-MD,arch/x86/crypto/.aesni-intel_asm.o.d -nostdinc -isystem /home/dileks/src/llvm-toolchain/install/lib/clang/10.0.1/include -I./arch/x86/include -I./arch/x86/include/generated  -I./include -I./arch/x86/include/uapi -I./arch/x86/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/kconfig.h -D__KERNEL__ -Qunused-arguments -D__ASSEMBLY__ -fno-PIE -Werror=unknown-warning-option -m64 -Wa,-gdwarf-2 -Wa,--compress-debug-sections=zlib -DCC_USING_FENTRY  -DMODULE  -c -o arch/x86/crypto/aesni-intel_asm.o arch/x86/crypto/aesni-intel_asm.S
arch/x86/crypto/aesni-intel_asm.S:47:1: warning: DWARF2 only supports one section per compilation unit
.section .rodata.cst16.gf128mul_x_ble_mask, "aM", @progbits, 16
^
arch/x86/crypto/aesni-intel_asm.S:51:1: warning: DWARF2 only supports one section per compilation unit
.section .rodata.cst16.POLY, "aM", @progbits, 16
^
arch/x86/crypto/aesni-intel_asm.S:54:1: warning: DWARF2 only supports one section per compilation unit
.section .rodata.cst16.TWOONE, "aM", @progbits, 16
^
arch/x86/crypto/aesni-intel_asm.S:58:1: warning: DWARF2 only supports one section per compilation unit
.section .rodata.cst16.SHUF_MASK, "aM", @progbits, 16
^
arch/x86/crypto/aesni-intel_asm.S:61:1: warning: DWARF2 only supports one section per compilation unit
.section .rodata.cst16.MASK1, "aM", @progbits, 16
^
arch/x86/crypto/aesni-intel_asm.S:64:1: warning: DWARF2 only supports one section per compilation unit
.section .rodata.cst16.MASK2, "aM", @progbits, 16
^
arch/x86/crypto/aesni-intel_asm.S:67:1: warning: DWARF2 only supports one section per compilation unit
.section .rodata.cst16.ONE, "aM", @progbits, 16
^
arch/x86/crypto/aesni-intel_asm.S:70:1: warning: DWARF2 only supports one section per compilation unit
.section .rodata.cst16.F_MIN_MASK, "aM", @progbits, 16
^
arch/x86/crypto/aesni-intel_asm.S:73:1: warning: DWARF2 only supports one section per compilation unit
.section .rodata.cst16.dec, "aM", @progbits, 16
^
arch/x86/crypto/aesni-intel_asm.S:76:1: warning: DWARF2 only supports one section per compilation unit
.section .rodata.cst16.enc, "aM", @progbits, 16
^
arch/x86/crypto/aesni-intel_asm.S:83:1: warning: DWARF2 only supports one section per compilation unit
.section .rodata, "a", @progbits
^
<instantiation>:15:74: error: too many positional arguments
 PRECOMPUTE 8*3+8(%rsp), %xmm1, %xmm2, %xmm3, %xmm4, %xmm5, %xmm6, %xmm7,
                                                                         ^
arch/x86/crypto/aesni-intel_asm.S:1598:2: note: while in macro instantiation
 GCM_INIT %r9, 8*3 +8(%rsp), 8*3 +16(%rsp), 8*3 +24(%rsp)
 ^
<instantiation>:47:2: error: unknown use of instruction mnemonic without a size suffix
 GHASH_4_ENCRYPT_4_PARALLEL_dec %xmm9, %xmm10, %xmm11, %xmm12, %xmm13, %xmm14, %xmm0, %xmm1, %xmm2, %xmm3, %xmm4, %xmm5, %xmm6, %xmm7, %xmm8, enc
 ^
arch/x86/crypto/aesni-intel_asm.S:1599:2: note: while in macro instantiation
 GCM_ENC_DEC dec
 ^
<instantiation>:15:74: error: too many positional arguments
 PRECOMPUTE 8*3+8(%rsp), %xmm1, %xmm2, %xmm3, %xmm4, %xmm5, %xmm6, %xmm7,
                                                                         ^
arch/x86/crypto/aesni-intel_asm.S:1686:2: note: while in macro instantiation
 GCM_INIT %r9, 8*3 +8(%rsp), 8*3 +16(%rsp), 8*3 +24(%rsp)
 ^
<instantiation>:47:2: error: unknown use of instruction mnemonic without a size suffix
 GHASH_4_ENCRYPT_4_PARALLEL_enc %xmm9, %xmm10, %xmm11, %xmm12, %xmm13, %xmm14, %xmm0, %xmm1, %xmm2, %xmm3, %xmm4, %xmm5, %xmm6, %xmm7, %xmm8, enc
 ^
arch/x86/crypto/aesni-intel_asm.S:1687:2: note: while in macro instantiation
 GCM_ENC_DEC enc
 ^
<instantiation>:15:67: error: too many positional arguments
 PRECOMPUTE %rcx, %xmm1, %xmm2, %xmm3, %xmm4, %xmm5, %xmm6, %xmm7,
                                                                  ^
arch/x86/crypto/aesni-intel_asm.S:1707:2: note: while in macro instantiation
 GCM_INIT %rdx, %rcx,%r8, %r9
 ^
<instantiation>:47:2: error: unknown use of instruction mnemonic without a size suffix
 GHASH_4_ENCRYPT_4_PARALLEL_enc %xmm9, %xmm10, %xmm11, %xmm12, %xmm13, %xmm14, %xmm0, %xmm1, %xmm2, %xmm3, %xmm4, %xmm5, %xmm6, %xmm7, %xmm8, enc
 ^
arch/x86/crypto/aesni-intel_asm.S:1722:2: note: while in macro instantiation
 GCM_ENC_DEC enc
 ^
<instantiation>:47:2: error: unknown use of instruction mnemonic without a size suffix
 GHASH_4_ENCRYPT_4_PARALLEL_dec %xmm9, %xmm10, %xmm11, %xmm12, %xmm13, %xmm14, %xmm0, %xmm1, %xmm2, %xmm3, %xmm4, %xmm5, %xmm6, %xmm7, %xmm8, enc
 ^
arch/x86/crypto/aesni-intel_asm.S:1737:2: note: while in macro instantiation
 GCM_ENC_DEC dec
 ^
<instantiation>:2:2: warning: DWARF2 only supports one section per compilation unit
 .pushsection .discard.ignore_alts
 ^
<instantiation>:2:2: note: while in macro instantiation
 ANNOTATE_IGNORE_ALTERNATIVE
 ^
arch/x86/crypto/aesni-intel_asm.S:2761:2: note: while in macro instantiation
 CALL_NOSPEC %r11
 ^
<instantiation>:2:2: warning: DWARF2 only supports one section per compilation unit
 .pushsection .discard.retpoline_safe
 ^
<instantiation>:2:2: note: while in macro instantiation
 ANNOTATE_RETPOLINE_SAFE; call *%r11
 ^
<instantiation>:3:2: note: while in macro instantiation
 ALTERNATIVE_2 "ANNOTATE_RETPOLINE_SAFE; call *%r11", "RETPOLINE_CALL %r11", ( 7*32+12), "lfence; ANNOTATE_RETPOLINE_SAFE; call *%r11", ( 7*32+13)
 ^
arch/x86/crypto/aesni-intel_asm.S:2761:2: note: while in macro instantiation
 CALL_NOSPEC %r11
 ^
<instantiation>:8:2: warning: DWARF2 only supports one section per compilation unit
 .pushsection .altinstructions,"a"
 ^
<instantiation>:3:2: note: while in macro instantiation
 ALTERNATIVE_2 "ANNOTATE_RETPOLINE_SAFE; call *%r11", "RETPOLINE_CALL %r11", ( 7*32+12), "lfence; ANNOTATE_RETPOLINE_SAFE; call *%r11", ( 7*32+13)
 ^
arch/x86/crypto/aesni-intel_asm.S:2761:2: note: while in macro instantiation
 CALL_NOSPEC %r11
 ^
<instantiation>:13:2: warning: DWARF2 only supports one section per compilation unit
 .pushsection .altinstr_replacement,"ax"
 ^
<instantiation>:3:2: note: while in macro instantiation
 ALTERNATIVE_2 "ANNOTATE_RETPOLINE_SAFE; call *%r11", "RETPOLINE_CALL %r11", ( 7*32+12), "lfence; ANNOTATE_RETPOLINE_SAFE; call *%r11", ( 7*32+13)
 ^
arch/x86/crypto/aesni-intel_asm.S:2761:2: note: while in macro instantiation
 CALL_NOSPEC %r11
 ^
make[5]: *** [scripts/Makefile.build:349: arch/x86/crypto/aesni-intel_asm.o] Error 1
make[4]: *** [scripts/Makefile.build:488: arch/x86/crypto] Error 2
make[4]: *** Waiting for unfinished jobs....

dileks · 2020-06-11T10:40:31Z

I try with:

scripts/config -e DEBUG_INFO_DWARF4

dileks · 2020-06-11T11:02:25Z

I tried with:

diff --git a/Makefile b/Makefile
index 812f4f1ee764..4d946b996e29 100644
--- a/Makefile
+++ b/Makefile
@@ -803,10 +803,12 @@ DEBUG_CFLAGS      += -gsplit-dwarf
 else
 DEBUG_CFLAGS   += -g
 endif
-KBUILD_AFLAGS  += -Wa,-gdwarf-2
-endif
 ifdef CONFIG_DEBUG_INFO_DWARF4
 DEBUG_CFLAGS   += -gdwarf-4
+KBUILD_AFLAGS  += -Wa,-gdwarf-4
+else
+KBUILD_AFLAGS  += -Wa,-gdwarf-2
+endif
 endif
 
 ifdef CONFIG_DEBUG_INFO_REDUCED

In my build-log I see:

  clang-10 -Wp,-MD,arch/x86/crypto/.aesni-intel_asm.o.d -nostdinc -isystem /home/dileks/src/llvm-toolchain/install/lib/clang/10.0.1/include -I./arch/x86/include -I./arch/x86/include/generated  -I./include -I./arch/x86/include/uapi -I./arch/x86/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/kconfig.h -D__KERNEL__ -Qunused-arguments -D__ASSEMBLY__ -fno-PIE -Werror=unknown-warning-option -m64 -Wa,-gdwarf-4 -Wa,--compress-debug-sections=zlib -DCC_USING_FENTRY  -DMODULE  -c -o arch/x86/crypto/aesni-intel_asm.o arch/x86/crypto/aesni-intel_asm.S
<instantiation>:15:74: error: too many positional arguments
 PRECOMPUTE 8*3+8(%rsp), %xmm1, %xmm2, %xmm3, %xmm4, %xmm5, %xmm6, %xmm7,
                                                                         ^
arch/x86/crypto/aesni-intel_asm.S:1598:2: note: while in macro instantiation
 GCM_INIT %r9, 8*3 +8(%rsp), 8*3 +16(%rsp), 8*3 +24(%rsp)
 ^
<instantiation>:47:2: error: unknown use of instruction mnemonic without a size suffix
 GHASH_4_ENCRYPT_4_PARALLEL_dec %xmm9, %xmm10, %xmm11, %xmm12, %xmm13, %xmm14, %xmm0, %xmm1, %xmm2, %xmm3, %xmm4, %xmm5, %xmm6, %xmm7, %xmm8, enc
 ^
arch/x86/crypto/aesni-intel_asm.S:1599:2: note: while in macro instantiation
 GCM_ENC_DEC dec
 ^
<instantiation>:15:74: error: too many positional arguments
 PRECOMPUTE 8*3+8(%rsp), %xmm1, %xmm2, %xmm3, %xmm4, %xmm5, %xmm6, %xmm7,
                                                                         ^
arch/x86/crypto/aesni-intel_asm.S:1686:2: note: while in macro instantiation
 GCM_INIT %r9, 8*3 +8(%rsp), 8*3 +16(%rsp), 8*3 +24(%rsp)
 ^
   ./tools/objtool/objtool orc generate  --no-fp --retpoline --uaccess kernel/user.o
  if objdump -h kernel/user.o | grep -q __ksymtab; then clang-10 -E -D__GENKSYMS__ -Wp,-MD,kernel/.user.o.d -nostdinc -isystem /home/dileks/src/llvm-toolchain/install/lib/clang/10.0.1/include -I./arch/x86/include -I./arch/x86/include/generated  -I./include -I./arch/x86/include/uapi -I./arch/x86/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/kconfig.h -include ./include/linux/compiler_types.h -D__KERNEL__ -Qunused-arguments -Wall -Wundef -Werror=strict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -fshort-wchar -fno-PIE -Werror=implicit-function-declaration -Werror=implicit-int -Wno-format-security -std=gnu89 -Werror=unknown-warning-option -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -mno-avx -m64 -mno-80387 -mstack-alignment=8 -mtune=generic -mno-red-zone -mcmodel=kernel -Wno-sign-compare -fno-asynchronous-unwind-tables -mretpoline-external-thunk -fno-delete-null-pointer-checks -Wno-address-of-packed-member -O2 -Wframe-larger-than=2048 -fstack-protector-strong -Wno-format-invalid-specifier -Wno-gnu -mno-global-merge -Wno-unused-const-variable -g -gdwarf-4 -gz=zlib -pg -mfentry -DCC_USING_FENTRY -Wdeclaration-after-statement -Wvla -Wno-pointer-sign -Wno-array-bounds -fno-strict-overflow -fno-merge-all-constants -fno-stack-check -Werror=date-time -Werror=incompatible-pointer-types -fmacro-prefix-map=./= -fcf-protection=none -Wno-initializer-overrides -Wno-format -Wno-sign-compare -Wno-format-zero-length -Wno-tautological-constant-out-of-range-compare    -DKBUILD_MODFILE='"kernel/user"' -DKBUILD_BASENAME='"user"' -DKBUILD_MODNAME='"user"' kernel/user.c | scripts/genksyms/genksyms    -r /dev/null > kernel/.tmp_user.ver; ld.lld-10 -m elf_x86_64  -z max-page-size=0x200000 --compress-debug-sections=zlib -r -o kernel/.tmp_user.o kernel/user.o -T kernel/.tmp_user.ver; mv -f kernel/.tmp_user.o kernel/user.o; rm -f kernel/.tmp_user.ver; fi
<instantiation>:47:2: error: unknown use of instruction mnemonic without a size suffix
 GHASH_4_ENCRYPT_4_PARALLEL_enc %xmm9, %xmm10, %xmm11, %xmm12, %xmm13, %xmm14, %xmm0, %xmm1, %xmm2, %xmm3, %xmm4, %xmm5, %xmm6, %xmm7, %xmm8, enc
 ^
arch/x86/crypto/aesni-intel_asm.S:1687:2: note: while in macro instantiation
 GCM_ENC_DEC enc
 ^
<instantiation>:15:67: error: too many positional arguments
 PRECOMPUTE %rcx, %xmm1, %xmm2, %xmm3, %xmm4, %xmm5, %xmm6, %xmm7,
                                                                  ^
arch/x86/crypto/aesni-intel_asm.S:1707:2: note: while in macro instantiation
 GCM_INIT %rdx, %rcx,%r8, %r9
 ^
<instantiation>:47:2: error: unknown use of instruction mnemonic without a size suffix
 GHASH_4_ENCRYPT_4_PARALLEL_enc %xmm9, %xmm10, %xmm11, %xmm12, %xmm13, %xmm14, %xmm0, %xmm1, %xmm2, %xmm3, %xmm4, %xmm5, %xmm6, %xmm7, %xmm8, enc
 ^
arch/x86/crypto/aesni-intel_asm.S:1722:2: note: while in macro instantiation
 GCM_ENC_DEC enc
 ^
<instantiation>:47:2: error: unknown use of instruction mnemonic without a size suffix
 GHASH_4_ENCRYPT_4_PARALLEL_dec %xmm9, %xmm10, %xmm11, %xmm12, %xmm13, %xmm14, %xmm0, %xmm1, %xmm2, %xmm3, %xmm4, %xmm5, %xmm6, %xmm7, %xmm8, enc
 ^
arch/x86/crypto/aesni-intel_asm.S:1737:2: note: while in macro instantiation
 GCM_ENC_DEC dec
 ^
make[5]: *** [scripts/Makefile.build:349: arch/x86/crypto/aesni-intel_asm.o] Error 1

dileks · 2020-06-11T13:21:50Z

I jumped to latest Linus Git and installed clang-11 and ld.lld-11 from <apt.llvm.org>:

$ git describe 
v5.7-14029-gb29482fde649

$ clang-11 --version
Debian clang version 11.0.0-++20200610100654+51a822724da-1~exp1~20200610201314.3290 
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

$ ld.lld-11 --version
LLD 11.0.0 (compatible with GNU linkers)

Instructions:

MAKE="make V=1" ; COMPILER="clang-11" ; LINKER="ld.lld-11" ; MAKE_OPTS="CC=$COMPILER HOSTCC=$COMPILER LD=$LINKER HOSTLD=$LINKER LLVM_IAS=1"

echo $MAKE $MAKE_OPTS
make V=1 CC=clang-11 HOSTCC=clang-11 LD=ld.lld-11 HOSTLD=ld.lld-11 LLVM_IAS=1

$MAKE $MAKE_OPTS defconfig

scripts/config -e CRYPTO_AES_NI_INTEL

scripts/config -s CRYPTO_AES_NI_INTEL
y

$MAKE $MAKE_OPTS arch/x86/crypto/aesni-intel_asm.o 2>&1 | tee ../make-log_clang-11_ld_lld-11_CRYPTO_AES_NI_INTEL-y_LLVM_IAS-1.txt

From my make-log.txt:

  clang-11 -Wp,-MMD,arch/x86/crypto/.aesni-intel_asm.o.d -nostdinc -isystem /usr/lib/llvm-11/lib/clang/11.0.0/include -I./arch/x86/include -I./arch/x86/include/generated  -I./include -I./arch/x86/include/uapi -I./arch/x86/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/kconfig.h -D__KERNEL__ -Qunused-arguments -D__ASSEMBLY__ -fno-PIE -Werror=unknown-warning-option -m64    -c -o arch/x86/crypto/aesni-intel_asm.o arch/x86/crypto/aesni-intel_asm.S
<instantiation>:15:74: error: too many positional arguments
 PRECOMPUTE 8*3+8(%rsp), %xmm1, %xmm2, %xmm3, %xmm4, %xmm5, %xmm6, %xmm7,
                                                                         ^
arch/x86/crypto/aesni-intel_asm.S:1598:2: note: while in macro instantiation
 GCM_INIT %r9, 8*3 +8(%rsp), 8*3 +16(%rsp), 8*3 +24(%rsp)
 ^
<instantiation>:47:2: error: unknown use of instruction mnemonic without a size suffix
 GHASH_4_ENCRYPT_4_PARALLEL_dec %xmm9, %xmm10, %xmm11, %xmm12, %xmm13, %xmm14, %xmm0, %xmm1, %xmm2, %xmm3, %xmm4, %xmm5, %xmm6, %xmm7, %xmm8, enc
 ^
arch/x86/crypto/aesni-intel_asm.S:1599:2: note: while in macro instantiation
 GCM_ENC_DEC dec
 ^
<instantiation>:15:74: error: too many positional arguments
 PRECOMPUTE 8*3+8(%rsp), %xmm1, %xmm2, %xmm3, %xmm4, %xmm5, %xmm6, %xmm7,
                                                                         ^
arch/x86/crypto/aesni-intel_asm.S:1686:2: note: while in macro instantiation
 GCM_INIT %r9, 8*3 +8(%rsp), 8*3 +16(%rsp), 8*3 +24(%rsp)
 ^
<instantiation>:47:2: error: unknown use of instruction mnemonic without a size suffix
 GHASH_4_ENCRYPT_4_PARALLEL_enc %xmm9, %xmm10, %xmm11, %xmm12, %xmm13, %xmm14, %xmm0, %xmm1, %xmm2, %xmm3, %xmm4, %xmm5, %xmm6, %xmm7, %xmm8, enc
 ^
arch/x86/crypto/aesni-intel_asm.S:1687:2: note: while in macro instantiation
 GCM_ENC_DEC enc
 ^
<instantiation>:15:67: error: too many positional arguments
 PRECOMPUTE %rcx, %xmm1, %xmm2, %xmm3, %xmm4, %xmm5, %xmm6, %xmm7,
                                                                  ^
arch/x86/crypto/aesni-intel_asm.S:1707:2: note: while in macro instantiation
 GCM_INIT %rdx, %rcx,%r8, %r9
 ^
<instantiation>:47:2: error: unknown use of instruction mnemonic without a size suffix
 GHASH_4_ENCRYPT_4_PARALLEL_enc %xmm9, %xmm10, %xmm11, %xmm12, %xmm13, %xmm14, %xmm0, %xmm1, %xmm2, %xmm3, %xmm4, %xmm5, %xmm6, %xmm7, %xmm8, enc
 ^
arch/x86/crypto/aesni-intel_asm.S:1722:2: note: while in macro instantiation
 GCM_ENC_DEC enc
 ^
<instantiation>:47:2: error: unknown use of instruction mnemonic without a size suffix
 GHASH_4_ENCRYPT_4_PARALLEL_dec %xmm9, %xmm10, %xmm11, %xmm12, %xmm13, %xmm14, %xmm0, %xmm1, %xmm2, %xmm3, %xmm4, %xmm5, %xmm6, %xmm7, %xmm8, enc
 ^
arch/x86/crypto/aesni-intel_asm.S:1737:2: note: while in macro instantiation
 GCM_ENC_DEC dec
 ^
make[2]: *** [scripts/Makefile.build:361: arch/x86/crypto/aesni-intel_asm.o] Error 1
make[1]: *** [scripts/Makefile.build:497: arch/x86/crypto] Error 2

dileks · 2020-06-11T13:37:30Z

Tried with adding LLVM=1 manually:

$ echo $MAKE $MAKE_OPTS
make V=1 CC=clang-11 LD=ld.lld-11 AR=llvm-ar-11 NM=llvm-nm-11 STRIP=llvm-strip-11 OBJCOPY=llvm-objcopy-11 OBJDUMP=llvm-objdump-11 OBJSIZE=llvm-size-11 READELF=llvm-readelf-11 HOSTCC=clang-11 HOSTCXX=clang++-11 HOSTAR=llvm-ar-11 HOSTLD=ld.lld-11 LLVM_IAS=1

The same BROKEN output like above.

dileks · 2020-06-11T13:38:33Z

@nickdesaulniers

With which version of llvm-toolchain did you had success with LLVM_IAS=1?

dileks · 2020-06-11T14:04:55Z

I will see how far I get with:

diff --git a/arch/x86/crypto/Makefile b/arch/x86/crypto/Makefile
--- a/arch/x86/crypto/Makefile
+++ b/arch/x86/crypto/Makefile
@@ -2,6 +2,8 @@
 #
 # x86 crypto algorithms
 
+KBUILD_CPPFLAGS += -no-integrated-as
+
 OBJECT_FILES_NON_STANDARD := y

dileks · 2020-06-11T15:02:33Z

I am building full DWARF4 now and my diff looks like this:

diff --git a/Makefile b/Makefile
index 812f4f1ee764..f37f393cdcf0 100644
--- a/Makefile
+++ b/Makefile
@@ -561,6 +561,10 @@ endif
 CLANG_FLAGS    += -Werror=unknown-warning-option
 KBUILD_CFLAGS  += $(CLANG_FLAGS)
 KBUILD_AFLAGS  += $(CLANG_FLAGS)
+ifdef CONFIG_DEBUG_INFO_DWARF4
+KBUILD_CFLAGS  += -gdwarf-4
+KBUILD_AFLAGS  += -Wa,-gdwarf-4
+endif
 export CLANG_FLAGS
 endif
 
@@ -803,10 +807,12 @@ DEBUG_CFLAGS      += -gsplit-dwarf
 else
 DEBUG_CFLAGS   += -g
 endif
-KBUILD_AFLAGS  += -Wa,-gdwarf-2
-endif
 ifdef CONFIG_DEBUG_INFO_DWARF4
 DEBUG_CFLAGS   += -gdwarf-4
+KBUILD_AFLAGS  += -Wa,-gdwarf-4
+else
+KBUILD_AFLAGS  += -Wa,-gdwarf-2
+endif
 endif
 
 ifdef CONFIG_DEBUG_INFO_REDUCED
diff --git a/arch/x86/crypto/Makefile b/arch/x86/crypto/Makefile
index a31de0c6ccde..a28810a6a489 100644
--- a/arch/x86/crypto/Makefile
+++ b/arch/x86/crypto/Makefile
@@ -2,6 +2,8 @@
 #
 # x86 crypto algorithms
 
+KBUILD_CPPFLAGS += -no-integrated-as
+
 OBJECT_FILES_NON_STANDARD := y
 
 obj-$(CONFIG_CRYPTO_GLUE_HELPER_X86) += glue_helper.o

dileks · 2020-06-11T15:09:04Z

@nickdesaulniers patch is queued up.

https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git/log/?h=queue/5.7

dileks · 2020-06-11T19:13:12Z

Cool, I was able to build with LLVM_IAS=1 and boot on bare metal:

$ cat start-build.txt 
dileks    374356  374337  0 16:45 pts/1    00:00:00 make V=1 CC=clang-10 HOSTCC=clang-10 LD=ld.lld-10 HOSTLD=ld.lld-10 LLVM_IAS=1 -j3 KBUILD_VERBOSE=1 KBUILD_BUILD_HOST=iniza KBUILD_BUILD_USER=sedat.dilek@gmail.com KBUILD_BUILD_TIMESTAMP=2020-06-11 LOCALVERSION=-3-amd64-clang bindeb-pkg KDEB_PKGVERSION=5.7.2-3~bullseye+dileks1

$ cat /proc/version 
Linux version 5.7.2-3-amd64-clang (sedat.dilek@gmail.com@iniza) (clang version 10.0.1 (https://github.com/llvm/llvm-project b6efa2365812f31667485c8948d49621ebf952f2), LLD 10.0.1 (https://github.com/llvm/llvm-project b6efa2365812f31667485c8948d49621ebf952f2)) #3~bullseye+dileks1 SMP 2020-06-11

I needed a 2nd hunk to previous diff:

diff --git a/arch/x86/lib/Makefile b/arch/x86/lib/Makefile
index 5246db42de45..af6cbb746cef 100644
--- a/arch/x86/lib/Makefile
+++ b/arch/x86/lib/Makefile
@@ -3,6 +3,8 @@
 # Makefile for x86 specific library files.
 #
 
+KBUILD_CPPFLAGS += -no-integrated-as
+
 # Produces uninteresting flaky coverage.
 KCOV_INSTRUMENT_delay.o        := n

En plus, I pulled from tip.git#tags/objtool-core-2020-06-01 and jpoimboe/linux.git#objtool/core to have a recent objtool.

I will see if I add CFLAGS_<objectfile> += -fno-integrated-as to arch/x86/crypto/aesni-intel_asm.o and arch/x86/lib/memcpy_64.o and see which files are really affected.

Sidenote: Without my wrapper-scripts to compiler (uses ccache) and linker ccache I could reduce the build-time by approx one hour.

nickdesaulniers · 2020-06-11T20:05:27Z

Thanks for the report. I've filed #1050 for the bug (requires additional config beyond defconfig).

For -Wa,-gdwarf-4, that looks good, similar to my suggestion. I plan to fix that up with my local patch set for dwarf-5 support, since I need to modify it anyway.

Without my wrapper-scripts to compiler (uses ccache) and linker ccache I could reduce the build-time by approx one hour.

If you're getting cache misses (new compiler, or not using a deterministic build timestamp) then ccache should be slower than no ccache. The hope is following builds that don't change too much become hotter and build faster.

Also #1043 should be a blocker here as we disabled LLVM_IAS in ClangBuiltLinux/continuous-integration#276 due to it.

Closing for now, we can still follow up here or in #1050

dileks · 2020-06-11T20:08:08Z

Unsetting above diff makes this visible Error 255:

   ./tools/objtool/objtool orc generate  --no-fp --retpoline --uaccess arch/x86/lib/memcpy_64.o

arch/x86/lib/memcpy_64.o: warning: objtool: memcpy_erms(): can't find starting instruction
make[4]: *** [scripts/Makefile.build:349: arch/x86/lib/memcpy_64.o] Error 255

   ./tools/objtool/objtool orc generate  --no-fp --retpoline --uaccess arch/x86/lib/memset_64.o

arch/x86/lib/memset_64.o: warning: objtool: memset_erms(): can't find starting instruction
make[4]: *** [scripts/Makefile.build:349: arch/x86/lib/memset_64.o] Error 255
make[4]: *** Deleting file 'arch/x86/lib/memset_64.o'
make[3]: *** [Makefile:1741: arch/x86/lib] Error 2

dileks · 2020-06-11T20:11:38Z

How can I see this is real DWARF4?

dileks · 2020-06-11T20:36:10Z

My latest diff:

diff --git a/Makefile b/Makefile
index 812f4f1ee764..68ae67cdde29 100644
--- a/Makefile
+++ b/Makefile
@@ -561,6 +561,10 @@ endif
 CLANG_FLAGS    += -Werror=unknown-warning-option
 KBUILD_CFLAGS  += $(CLANG_FLAGS)
 KBUILD_AFLAGS  += $(CLANG_FLAGS)
+ifdef LLVM_IAS
+KBUILD_CFLAGS  += -gdwarf-4
+KBUILD_AFLAGS  += -Wa,-gdwarf-4
+endif
 export CLANG_FLAGS
 endif
 
@@ -803,10 +807,15 @@ DEBUG_CFLAGS      += -gsplit-dwarf
 else
 DEBUG_CFLAGS   += -g
 endif
+ifndef LLVM_IAS
 KBUILD_AFLAGS  += -Wa,-gdwarf-2
 endif
+endif
 ifdef CONFIG_DEBUG_INFO_DWARF4
 DEBUG_CFLAGS   += -gdwarf-4
+ifdef LLVM_IAS
+KBUILD_AFLAGS  += -Wa,-gdwarf-4
+endif
 endif
 
 ifdef CONFIG_DEBUG_INFO_REDUCED
diff --git a/arch/x86/crypto/Makefile b/arch/x86/crypto/Makefile
index a31de0c6ccde..bd5be603fdf3 100644
--- a/arch/x86/crypto/Makefile
+++ b/arch/x86/crypto/Makefile
@@ -2,6 +2,10 @@
 #
 # x86 crypto algorithms
 
+ifdef LLVM_IAS
+KBUILD_CPPFLAGS += -no-integrated-as
+endif
+
 OBJECT_FILES_NON_STANDARD := y
 
 obj-$(CONFIG_CRYPTO_GLUE_HELPER_X86) += glue_helper.o
diff --git a/arch/x86/lib/Makefile b/arch/x86/lib/Makefile
index 5246db42de45..619107bec0d9 100644
--- a/arch/x86/lib/Makefile
+++ b/arch/x86/lib/Makefile
@@ -3,6 +3,10 @@
 # Makefile for x86 specific library files.
 #
 
+ifdef LLVM_IAS
+KBUILD_CPPFLAGS += -no-integrated-as
+endif
+
 # Produces uninteresting flaky coverage.
 KCOV_INSTRUMENT_delay.o        := n

nickdesaulniers · 2020-06-12T04:39:53Z

How can I see this is real DWARF4?

llvm-dwarfdump will print the version as one of very first few lines it prints. Not easy to see, it in a list, IIRC.

dileks · 2020-06-12T04:53:26Z

Thanks.

I see with llvm-dwarfdump DWARF version = 0x0004 (here: vmlinux.o)

$ llvm-dwarfdump vmlinux.o | head -5
vmlinux.o:      file format ELF64-x86-64

.debug_info contents:
0x00000000: Compile Unit: length = 0x000003a3 version = 0x0004 abbr_offset = 0x0000 addr_size = 0x08 (next unit at 0x000003a7)

dileks · 2020-06-12T16:59:32Z

@masahir0y commented in the thread [1]:

"x86/crypto: Set -no-integrated-as for specific object-file when building with LLVM_IAS=1"

The source file is .S (assembly file),
so

AFLAGS_aes_ctrby8_avx-x86_64.o += -no-integrated-as

Or,

asflags-y += -no-integrated-as

, which is effective for all .S files in the directory.

Here you have another example I use to dig into LLVM_IAS=1 issues in arch/x86/lib:

diff --git a/arch/x86/lib/Makefile b/arch/x86/lib/Makefile
index 619107bec0d9..2c07cc651d52 100644
--- a/arch/x86/lib/Makefile
+++ b/arch/x86/lib/Makefile
@@ -3,10 +3,6 @@
 # Makefile for x86 specific library files.
 #
 
-ifdef LLVM_IAS
-KBUILD_CPPFLAGS += -no-integrated-as
-endif
-
 # Produces uninteresting flaky coverage.
 KCOV_INSTRUMENT_delay.o        := n
 
@@ -39,6 +35,9 @@ obj-$(CONFIG_SMP) += msr-smp.o cache-smp.o
 lib-y := delay.o misc.o cmdline.o cpu.o
 lib-y += usercopy_$(BITS).o usercopy.o getuser.o putuser.o
 lib-y += memcpy_$(BITS).o
+ifdef LLVM_IAS
+AFLAGS_memcpy_$(BITS).o += -no-integrated-as
+endif
 lib-$(CONFIG_INSTRUCTION_DECODER) += insn.o inat.o insn-eval.o
 lib-$(CONFIG_RANDOMIZE_BASE) += kaslr.o
 lib-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o
@@ -62,6 +61,9 @@ else
         lib-y += csum-partial_64.o csum-copy_64.o csum-wrappers_64.o
         lib-y += clear_page_64.o copy_page_64.o
         lib-y += memmove_64.o memset_64.o
+        ifndef LLVM_IAS
+        AFLAGS_memset_64.o += -no-integrated-as
+        endif
         lib-y += copy_user_64.o
        lib-y += cmpxchg16b_emu.o
 endif

[1] https://lore.kernel.org/r/CAK7LNAQuiuj5UifVBYEN7Xkp5GH0RNiWc5F3VyA1BAjGAUhqhw@mail.gmail.com/
[2] https://lore.kernel.org/r/CA+icZUX20_yarSs7fJWq6Sxy3xBaeUXSQjmMbjcQFXB4JnyijA@mail.gmail.com/

dileks · 2020-06-12T17:17:30Z

@nickdesaulniers

Can you comment on my DWARF-4 settings?
I have seen doubles in the make-lines.

Does KBUILD_AFLAGS += -Wa,-gdwarf-4 make sense to you?
Is this only useful in the DEBUG section of the toplevel Makefile?

IMHO the settings in the first block can go away?
I wanted to force DWARF-4 in the complete build.

What about the second block?

[ Makefile ]

ifneq ($(shell $(CC) --version 2>&1 | head -n 1 | grep clang),)
ifneq ($(CROSS_COMPILE),)
CLANG_FLAGS	+= --target=$(notdir $(CROSS_COMPILE:%-=%))
GCC_TOOLCHAIN_DIR := $(dir $(shell which $(CROSS_COMPILE)elfedit))
CLANG_FLAGS	+= --prefix=$(GCC_TOOLCHAIN_DIR)
GCC_TOOLCHAIN	:= $(realpath $(GCC_TOOLCHAIN_DIR)/..)
endif
ifneq ($(GCC_TOOLCHAIN),)
CLANG_FLAGS	+= --gcc-toolchain=$(GCC_TOOLCHAIN)
endif
ifneq ($(LLVM_IAS),1)
CLANG_FLAGS	+= -no-integrated-as
endif
CLANG_FLAGS	+= -Werror=unknown-warning-option
KBUILD_CFLAGS	+= $(CLANG_FLAGS)
KBUILD_AFLAGS	+= $(CLANG_FLAGS)
ifdef LLVM_IAS
KBUILD_CFLAGS	+= -gdwarf-4
KBUILD_AFLAGS	+= -Wa,-gdwarf-4
endif
export CLANG_FLAGS
endif

...

ifdef CONFIG_DEBUG_INFO
ifdef CONFIG_DEBUG_INFO_SPLIT
DEBUG_CFLAGS	+= -gsplit-dwarf
else
DEBUG_CFLAGS	+= -g
endif
ifndef LLVM_IAS
KBUILD_AFLAGS	+= -Wa,-gdwarf-2
endif
endif
ifdef CONFIG_DEBUG_INFO_DWARF4
DEBUG_CFLAGS	+= -gdwarf-4
ifdef LLVM_IAS
KBUILD_AFLAGS	+= -Wa,-gdwarf-4
endif
endif

dileks · 2020-06-12T17:33:45Z

3.10 Options for Debugging Your Program [1] says:

-g

Produce debugging information in the operating system’s native format (stabs, COFF, XCOFF, or DWARF). GDB can work with this debugging information.

On most systems that use stabs format, -g enables use of extra debugging information that only GDB can use; this extra information makes debugging work better in GDB but probably makes other debuggers crash or refuse to read the program. If you want to control for certain whether to generate the extra information, use -gstabs+, -gstabs, -gxcoff+, -gxcoff, or -gvms (see below).

-gdwarf
-gdwarf-version

Produce debugging information in DWARF format (if that is supported). The value of version may be either 2, 3, 4 or 5; the default version for most targets is 4. DWARF Version 5 is only experimental.

Note that with DWARF Version 2, some ports require and always use some non-conflicting DWARF 3 extensions in the unwind tables.

Version 4 may require GDB 7.0 and -fvar-tracking-assignments for maximum benefit.

GCC no longer supports DWARF Version 1, which is substantially different than Version 2 and later. For historical reasons, some other DWARF-related options such as -fno-dwarf2-cfi-asm) retain a reference to DWARF Version 2 in their names, but apply to all currently-supported versions of DWARF.

[1] https://gcc.gnu.org/onlinedocs/gcc/Debugging-Options.html

E5ten · 2020-06-13T20:29:50Z

With the patches to fix syntax in arch/x86/crypto/aes{ni-intel_asm,_ctrby8_avx-x86_64}.S, the patch to switch the use of movzxw->movzwq, and individually adding CFLAGS_initramfs.o += -no-integrated-as to init/Makefile and CFLAGS_elfcore.o += -no-integrated-as to kernel/Makefile to avoid #981, I can build an x86 kernel with LLVM_IAS=1, so once those changes are upstreamed and the linked issue is solved, we should be able to build an x86 kernel with integrated-as.

dileks · 2020-06-14T09:34:21Z

[ REPORT 2020-06-14 by dileks ]

[ DISCLAIMER ]

I collect here my experiences and want to share them with you.
Please do not close this issue - thanks.

THIS IS STILL EXPERIMENTAL - USE AT YOUR OWN RISK!

My work is in collaboration with ClangBuiltLinux folks.

[ MY PERSONAL DISCLAIMER ]

To say with the words of the author of curl and DOH (keynote @ FOSDEM 2019):
I do this for me - It has to work for me - in my environment - *first*.

Second, even experimental work has to boot on bare metal - my bare metal.

It has to build in my environment using latest "stable" Linux-Kernel and "stable" llvm-toolchain.

Debian/testing AMD64 with daily APT upgrades (plus selected packages from Debian/unstable)
Linux-kernel: v5.7.2 + queue-5.7 2020-06-11 includes @nickdesaulniers elfnote patch
llvm-toolchain: releas/10.x Git up to commit 2dc664d578f0e9c8ea5975eed745e322fa77bffe
apt.llvm.org: Debian packages version 1:10.0.1~++20200612013218+2dc664d578f-1~~exp1~~20200611233837.179

[ STATUS ]

This is the status from 14-Jun-2020.

[ LLVM_IAS PATCHSET ]

PREREQ: @nickdesaulniers elfnote patch (in Linus tree and linux-stable-rc.git#queue-5.7)
Explicitly use DWARF version 4: CONFIG_DEBUG_INFO_DWARF4=y, /me has set assembler-option for dwarf4 when CONFIG_DEBUG_INFO_DWARF4=y
make: Use clang-10 compiler and ld.lld-10 linker plus LLVM_IAS=1 and new OBJDUMP=llvm-objdump (@arndb patch)
x86/crypto: All solved - with known patches or diffs from @nickdesaulniers, @tpimh, @arndb, @jcai19 and @dileks (with suggested-by @topperc)
x86/lib: Workaround objtool errors in memcpy_64 and memset_64 with -no-integrated-as assembler-option (thanks @masahir0y)

NOTE: Contacted Linux/objtool maintainers: Peter Zijlstra is inspecting my .o files
NOTE: Dropped objtool stuff from upstream-5.8 + jpoimboe-Git (did not help - still see the same objtool errors)

[ KNOWN ISSUES @dileks ]

[ MINIMAL PATCHSET ]

XXX: TODO: Sort out relevant patches only
XXX: TODO: Create patches for upstream - with help of CBL folks - respect credits
XXX: TODO: Collect feedback - the more the better - Thanks @E5ten

NOTE: I started to label patches with "llvm-ias:" and/or add "when building with LLVM_IAS=1" which is IMHO too long.

[ SELFMADE TOOLCHAIN VS. APT.LLVM.ORG ]

NOTE: Same version!

XXX: TODO: Uninstall/Disable selfmade toolchain and use the Debian packages only - strategy?

[ KERN.LOG - c-index-test ]

When building a selfmade toolchain I see with c-index-test:

traps: clang[...] trap invalid opcode ip:[...] sp:[...] error:0 in clang-10[...+...]

XXX: REPRODUCIBLE

[ ZSTD-INITRAMFS ]

When creating a new initramfs with zstd-support under a LLVM_IAS+llvm-objdump kernel I see:

Message from syslogd@iniza at Jun 14 10:21:43 ...
kernel:[ 2427.576669] traps: PANIC: double fault, error_code: 0x0

REPRODUCER:
root# update-initramfs -c -k $(uname -r) -v 2>&1 | tee log_update-initramfs_$(uname -r).txt

XXX: REPRODUCIBLE

XXX: TODO: post syslog_20200614.txt (Call Trace: <ENTRY_TRAMPOLINE>)

[ crypto/algapi.c - crypto_wait_for_test ]

WARNING: CPU: 2 PID: 255 at crypto/algapi.c:404 crypto_wait_for_test

NOTE: I have some crypto-algo kconfigs enabled as modules.

Tried to fix with cherry-picked upstream commits:

beeb460 ("crypto: algapi - Avoid spurious modprobe on LOADED")
ec6e2bf ("crypto: algapi - create function to add request in front of queue")

XXX: REPRODUCIBLE (seen on each boot; upstream-5.8 patches did not help)

XXX: TODO: Contact Linux/crypto maintainers
XXX: TODO: post BROKEN_dmesg-x86-crypto-llvm_ias-1.txt

[ KDE/PLASMA ]

See several crashes "signald" shown in a infobox (see above zstd/initramfs etc.)

XXX: REPRODUCIBLE

[ CBL ISSUE-TRACKER ]

LINK: #1049
LINK: #1050
LINK: #1010
LINK: #1008
LINK: https://github.com/ClangBuiltLinux/continuous-integration/blob/master/patches/llvm-all/linux-next/arm64/silence-dwarf2-warnings.patch

[ THANKS ]

Thanks for using llvm-toolchain to build the Linux-kernel and your time and patience to answer my question.

My special thank-you to (in no preferential order):
@nickdesaulniers, @tpimh, @arndb, @jcai19, @topperc, @masahir0y, @nathanchance, @E5ten

Of course not to forget the people behind cbl/continuous-ci and all contributors to llvm/clang/lld/llvm-tools...
...and the Linux/x86 coding monsters.

[ YOUR FEEDBACK ]

You are welcome.

dileks · 2020-06-14T09:55:17Z

This patchset is from my for-5.7/x86-llvm-ias-dileks-v3 Git branch (on top of Linux v5.7.2):

[PATCH] kbuild: llvm-ias: Silence dwarf-2 warning
[PATCH] kbuild: llvm-ias: Add -Wa,-gdwarf-4 assembler option when CONFIG_DEBUG_INFO_DWARF4=y
[PATCH] x86/crypto: aesni-intel: Fix build with LLVM_IAS=1
[PATCH] x86/64/crypto: aesni-intel: Fix build with LLVM_IAS=1
[PATCH] x86/64/crypto: crc32c-intel: Fix build with LLVM_IAS=1
[PATCH] x86/lib: memcpy_64: Build with -no-integrated-as assembler option when LLVM_IAS=1
[PATCH] x86/lib: memset_64: Build with -no-integrated-as assembler option when LLVM_IAS=1

NOTE: This patchset does not respect any credits yet - take it as-is - status: WIP.

Feedback is welcome.

@E5ten
Can you test with this patchset?
As said you need the patch from [1].

[1] https://git.kernel.org/linus/51da9dfb7f20911ae4e79e9b412a9c2d4c373d4b
[2] https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git/log/?h=queue/5.7
[3] https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git/commit/?h=queue/5.7&id=e5aa55eb3a10d354eb16af98a73bb448af623abf

0001-kbuild-llvm-ias-Silence-dwarf-2-warning_patch.txt
0002-kbuild-llvm-ias-Add-Wa-gdwarf-4-assembler-option-whe_patch.txt
0003-x86-crypto-aesni-intel-Fix-build-with-LLVM_IAS-1_patch.txt
0004-x86-64-crypto-aesni-intel-Fix-build-with-LLVM_IAS-1_patch.txt
0005-x86-64-crypto-crc32c-intel-Fix-build-with-LLVM_IAS-1_patch.txt
0006-x86-lib-memcpy_64-Build-with-no-integrated-as-assemb_patch.txt
0007-x86-lib-memset_64-Build-with-no-integrated-as-assemb_patch.txt

dileks · 2020-06-14T10:05:36Z

Add my latest Linux-kernel config.

config-5.7.2-7-amd64-clang.txt

dileks · 2020-06-14T10:17:10Z

Maybe this is shorter (commit-subject):

kbuild: Silence dwarf-2 warning when LLVM_IAS=1
kbuild: Add dwarf-4 assembler option when LLVM_IAS=1
x86/crypto: aesni: Fix build with LLVM_IAS=1
x86/64/crypto: aesni: Fix build with LLVM_IAS=1
x86/64/crypto: crc32c: Fix build with LLVM_IAS=1
x86/lib: memcpy_64: Build with -no-integrated-as when LLVM_IAS=1
x86/lib: memset_64: Build with -no-integrated-as when LLVM_IAS=1

UPDATE: Add patchset v5

0001-kbuild-Silence-dwarf-2-warning-when-LLVM_IAS-1_patch.txt
0002-kbuild-Add-dwarf-4-assembler-option-when-LLVM_IAS-1_patch.txt
0003-x86-crypto-aesni-Fix-build-with-LLVM_IAS-1_patch.txt
0004-x86-64-crypto-aesni-Fix-build-with-LLVM_IAS-1_patch.txt
0005-x86-64-crypto-crc32c-Fix-build-with-LLVM_IAS-1_patch.txt
0006-x86-lib-memcpy_64-Build-with-no-integrated-as-when-LLVM_IAS-1_patch.txt
0007-x86-lib-memset_64-Build-with-no-integrated-as-when-LLVM_IAS-1_patch.txt

dileks · 2020-06-14T10:47:41Z

CFLAGS_initramfs.o += -no-integrated-as

According to @masahir0y this should be:

AFLAGS_initramfs.o += -no-integrated-as

Why did you need this?
What did you try to fix?

UPDATE: @E5ten you are right - it is an .o file not .S file (CFLAGS_initramfs.o line is correct).

dileks · 2020-06-15T16:36:10Z

@E5ten

Linux?
Today, I saw Linux v5.8-rc1 was released.

E5ten · 2020-06-15T16:36:20Z

5.7.2

dileks · 2020-06-15T16:38:20Z

@E5ten

You see no problems with the diff from #1008?

dileks · 2020-06-15T16:50:59Z

I switched over to latest known GOOD_REVISION = '8a5aea7b50429cd4a459511286a7a9f1a7f4f5e2' in tc-build.

tc-build says:

HEAD is now at 8a5aea7b5042 [X86][AVX] Fold extract_subvector(subv_broadcast(x),c) -> (x)

[1] https://github.com/ClangBuiltLinux/tc-build/blob/master/build-llvm.py#L20
[2] https://github.com/ClangBuiltLinux/tc-build/blob/master/build-llvm.py#L269

dileks · 2020-06-15T20:37:41Z

@E5ten

With llvm Git 8a5aea7b5042 memcpy_64/memset_64 is OK.

$ clang --version
clang version 11.0.0 (https://github.com/llvm/llvm-project 8a5aea7b50429cd4a459511286a7a9f1a7f4f5e2)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/dileks/src/llvm-toolchain/install/bin

$ ld.lld --version
LLD 11.0.0 (https://github.com/llvm/llvm-project 8a5aea7b50429cd4a459511286a7a9f1a7f4f5e2) (compatible with GNU linkers)

dileks · 2020-06-15T20:53:22Z

Ah I see CC_HAS_ASM_INLINE y with clang-11:

$ scripts/diffconfig /boot/config-5.7.2-1-amd64-clang .config
 BUILD_SALT "5.7.2-1-amd64-clang" -> "5.7.2-12-amd64-clang"
 CLANG_VERSION 100001 -> 110000
 DEBUG_INFO_DWARF4 n -> y
+CC_HAS_ASM_INLINE y
+TOOLS_SUPPORT_RELR y

dileks · 2020-06-16T06:06:34Z

I build with LLVM=1 and LLVM_IAS=1 using llvm-toolchain Git 8a5aea7b50429cd4a459511286a7a9f1a7f4f5e2 and this patchset:

elfnote: mark all .note sections SHF_ALLOC

kbuild: Silence dwarf-2 warning when LLVM_IAS=1
kbuild: Add dwarf-4 assembler option when LLVM_IAS=1
x86/crypto: aesni: Fix build with LLVM_IAS=1
x86/64/crypto: crc32c: Fix build with LLVM_IAS=1
x86/64/crypto: aesni: aes_ctrby8: Build with -no-integrated-as when LLVM_IAS=1

init: initramfs: Build with -no-integrated-as when LLVM=1 and LLVM_IAS=1
kernel: elfcore: Build with -no-integrated-as when LLVM=1 and LLVM_IAS=1

dileks · 2020-06-16T10:53:27Z

Looks good with above patchset.

root@iniza:~# cat /proc/version 
Linux version 5.7.2-13-amd64-clang (sedat.dilek@gmail.com@iniza) (clang version 11.0.0 (https://github.com/llvm/llvm-project 8a5aea7b50429cd4a459511286a7a9f1a7f4f5e2), LLD 11.0.0 (https://github.com/llvm/llvm-project 8a5aea7b50429cd4a459511286a7a9f1a7f4f5e2)) #13~bullseye+dileks1 SMP 2020-06-16

The only warning I see in dmesg:

root@iniza:~# LC_ALL=C dmesg -T | egrep -i 'alg:|aes|crc32c|crypt' | egrep -i -v 'fscrypt|systemd'
[Tue Jun 16 12:15:24 2020] cryptd: max_cpu_qlen set to 1000
[Tue Jun 16 12:15:25 2020] AES CTR mode by8 optimization enabled
[Tue Jun 16 12:15:25 2020] alg: aead: rfc4106-gcm-aesni encryption test failed (wrong result) on test vector 0, cfg="two even aligned splits"
[Tue Jun 16 12:15:25 2020] alg: aead: generic-gcm-aesni encryption test failed (wrong result) on test vector 1, cfg="two even aligned splits"
[Tue Jun 16 12:15:25 2020] alg: No test for fips(ansi_cprng) (fips_ansi_cprng)
[Tue Jun 16 12:25:45 2020] Btrfs loaded, crc32c=crc32c-intel

Check Linux Git:

$ cd /path/to/linux/git
$ git grep -E 'generic-gcm-aesni|rfc4106-gcm-aesni' arch/x86/crypto/
arch/x86/crypto/aesni-intel_glue.c:             .cra_driver_name        = "__rfc4106-gcm-aesni",
arch/x86/crypto/aesni-intel_glue.c:             .cra_driver_name        = "__generic-gcm-aesni",

Loaded kernel-modules:

root@iniza:~# lsmod | egrep 'aes|crc32c|crypt' | sort
aesni_intel           368640  2
crc32c_generic         16384  0
crc32c_intel           24576  5
cryptd                 28672  2 crypto_simd,ghash_clmulni_intel
crypto_simd            16384  1 aesni_intel
glue_helper            16384  1 aesni_intel
libaes                 20480  2 bluetooth,aesni_intel
libcrc32c              16384  2 btrfs,xfs

The problems when re-running update-initramfs are gone with switching to tc-build/GOOD_REVISION.

dileks · 2020-06-17T04:51:33Z

For Linux v5.7.3-rc2 I switched back to make CC=clang-11 HOSTCC=clang-11 LD=ld.lld-11 HOSTLD=ld.lld-11 OBJDUMP=llvm-objdump LLVM_IAS=1 and dropped LLVM=1 and correspondig patches:

init: initramfs: Build with -no-integrated-as when LLVM=1 and LLVM_IAS=1
kernel: elfcore: Build with -no-integrated-as when LLVM=1 and LLVM_IAS=1

dileks · 2020-06-17T14:15:41Z

Just want to point out when you build with LLVM_IAS=1 and have forced DWARF version 4 assembler-option in the toplevel Makefile and do workaround with -no-integrated-as like for known broken aes_ctrby8_avx-x86_64.o on my system you assemble with GNU/as and DWARF version 2 is produced:

# Set CONFIG_DEBUG_INFO_DWARF4=y
cd /path/to/linux/git
scripts/config -e DEBUG_INFO_DWARF4

[ Makefile ]

ifdef CONFIG_DEBUG_INFO
ifdef CONFIG_DEBUG_INFO_SPLIT
DEBUG_CFLAGS	+= -gsplit-dwarf
else
DEBUG_CFLAGS	+= -g
endif
ifndef LLVM_IAS
KBUILD_AFLAGS	+= -Wa,-gdwarf-2
endif
endif
ifdef CONFIG_DEBUG_INFO_DWARF4
DEBUG_CFLAGS	+= -gdwarf-4
ifdef LLVM_IAS
KBUILD_AFLAGS  += -Wa,-gdwarf-4
endif
endif

[ arch/x86/crypto/Makefile ]

obj-$(CONFIG_CRYPTO_AES_NI_INTEL) += aesni-intel.o
aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o
aesni-intel-$(CONFIG_64BIT) += aesni-intel_avx-x86_64.o aes_ctrby8_avx-x86_64.o
ifdef LLVM_IAS
AFLAGS_aes_ctrby8_avx-x86_64.o += -no-integrated-as
endif

$ llvm-dwarfdump arch/x86/crypto/aes_ctrby8_avx-x86_64.o
arch/x86/crypto/aes_ctrby8_avx-x86_64.o:        file format elf64-x86-64

.debug_info contents:
0x00000000: Compile Unit: length = 0x0000002a version = 0x0002 abbr_offset = 0x0000 addr_size = 0x08 (next unit at 0x0000002e)

0x0000000b: DW_TAG_compile_unit
              DW_AT_stmt_list   (0x00000000)
              DW_AT_low_pc      (0x0000000000000000)
              DW_AT_high_pc     (0x0000000000003cb2)
              DW_AT_name        ("arch/x86/crypto/aes_ctrby8_avx-x86_64.S")
              DW_AT_comp_dir    ("/home/dileks/src/linux-kernel/git")
              DW_AT_producer    ("GNU AS 2.34")
              DW_AT_language    (DW_LANG_Mips_Assembler)

$ llvm-dwarfdump arch/x86/crypto/aesni-intel_asm.o | head -15
arch/x86/crypto/aesni-intel_asm.o:      file format elf64-x86-64

.debug_info contents:
0x00000000: Compile Unit: length = 0x00002ad3 version = 0x0004 abbr_offset = 0x0000 addr_size = 0x08 (next unit at 0x00002ad7)

0x0000000b: DW_TAG_compile_unit
              DW_AT_stmt_list   (0x00000000)
              DW_AT_ranges      (0x00000000
                 [0x0000000000000000, 0x000000000000891f)
                 [0x0000000000000000, 0x000000000000003c))
              DW_AT_name        ("arch/x86/crypto/aesni-intel_asm.S")
              DW_AT_comp_dir    ("/home/dileks/src/linux-kernel/git")
              DW_AT_producer    ("clang version 11.0.0 (https://github.com/llvm/llvm-project 8da5b9083691b557f50f72ab099598bb291aec5f)")
              DW_AT_language    (DW_LANG_Mips_Assembler)

NOTE: See version = 0x0002 VS. version = 0x0004

UPDATE: 2020-06-18: Add Makefile excerpt and CONFIG_DEBUG_INFO_DWARF4=y

dileks · 2020-06-17T16:44:44Z

Linux v5.7.3 released - rebuild with latest patchset.

dileks · 2020-06-22T16:58:10Z

Progress on #1008 (comment)

This means I do not need any -no-integrated-as workaround to build/assemble with LLVM_IAS=1.

dileks · 2020-06-23T07:43:39Z

Patch submitted for #1008

[1] #1008 (comment)

dileks · 2020-06-23T07:44:33Z

Patch submitted for #1050

[1] #1050 (comment)

dileks · 2020-06-30T10:30:28Z

I have re-tested with LLVM v10.0.1-rc2 and Linux v5.7.6 and LLVM_IAS=1.

This needs the no-integrated-as workarounds for memcpy_64 and memset_64.

Furthermore, when I upgrade my initrd-image with ZSTD support I still see the kernel double-faults.

Conclusion:
With Linux v5.7.7 I will switch back to LLVM v11.0.1-git.
(See tc-build: GOOD_REVISION=8a5aea7b50429cd4a459511286a7a9f1a7f4f5e2)

P.S.: With dropping my DWARF-4 assembler option patch I lose the information in the .o files (.S file as a base) when checking with llvm-dwarfdump.

dileks · 2020-07-10T10:28:01Z

This (last) Monday with the official release of Linux v5.8-rc4 I switched over from Linux v5.7.y.

Inspired by @paulmenzel report in #1076 I switched over my llvm-toolchain to version 1:11~++20200701093119+ffee8040534-1~exp1 from Debian/experimental.

We have accepted two patches from @jcai19 (see #1008) and @dileks (see #1050) in crypto-2.6.git:

"crypto: aesni - add compatibility with IAS"
"crypto: aesni - Fix build with LLVM_IAS=1"

The last patch to be upstreamed and be full compliant with LLVM_IAS=1 is the patch from @arndb (see #1010).

This means we are (will be) able to set and use on x86-64:

CONFIG_CRYPTO_AES_NI_INTEL={y,m}
CONFIG_CRYPTO_CRC32C_INTEL={y,m}

Just as a sidenote:
I build and boot on bare metail with a manually setup LLVM=1 means with LLVM/utilities from Debian/experimental.

Note:
One workaround I have with -no-integrated-as is for arch/x86/entry/entry_64.S (@jcai19 offered a patch in #1043).
Feel free to test and report - I just compile/assemble-tested.

[1] #1076
[2] https://git.kernel.org/pub/scm/linux/kernel/git/herbert/cryptodev-2.6.git/commit/?id=44069737ac9625a0f02f0f7f5ab96aae4cd819bc
[3] https://git.kernel.org/pub/scm/linux/kernel/git/herbert/cryptodev-2.6.git/commit/?id=3347c8a079d67af21760a78cc5f2abbcf06d9571
[4] https://lore.kernel.org/all/CA+icZUWQVrph_Rv9O=Q4Fh8Y0jXzkWGCHN5DFAaCZeE5eXoPbw@mail.gmail.com/
[5] https://lore.kernel.org/all/CA+icZUXj4TKEduA5A0fpdvha6E2fV6aZXOZ8BXvD5G4eYBHqsg@mail.gmail.com/
[6] https://llvm.org/pr24494
[7] #1043

dileks · 2020-07-12T22:16:11Z

The workaround for arch/x86/entry/entry_64.o is no more needed with the diff #1043 (comment) from @jcai19 and my diff #1088 (comment).

So a complete build with LLVM_IAS=1 on Linux v5.8-rc4 is OK.

dileks · 2020-07-22T18:18:42Z

To build Linux version 5.8 with LLVM toolchain version 11.0.0-rc1 and LLVM_IAS=1 (and LLVM=1) I have these patches applied:

kbuild: Silence dwarf-2 warning when LLVM_IAS=1
Original-by: tpimh
Link: https://github.com/ClangBuiltLinux/continuous-integration/blob/master/patches/llvm-all/linux-next/arm64/silence-dwarf2-warnings.patch
kbuild: Add dwarf-4 assembler option when LLVM_IAS=1
Link: Status: Building with Clang's integrated assembler aka LLVM_IAS=1 #1049 (comment)
crypto: aesni - add compatibility with IAS
Link: https://git.kernel.org/linus/44069737ac9625a0f02f0f7f5ab96aae4cd819bc
crypto: aesni - Fix build with LLVM_IAS=1
Link: https://git.kernel.org/linus/3347c8a079d67af21760a78cc5f2abbcf06d9571
crypto: x86/crc32c - fix building with clang ias
Link: https://git.kernel.org/linus/44623b2818f4a442726639572f44fd9b6d0ef68c
x86: work around clang IAS bug referencing __force_order
Link: https://lore.kernel.org/patchwork/patch/1248382/

NOTE to 1. and 2.: DWARF-4 seems to be a good choice (sidenote: binutils >= 2.35 has support for it)

NOTE to 2.: CONFIG_DEBUG_INFO_DWARF4=y

NOTE to 3. and 4.: CONFIG_CRYPTO_AES_NI_INTEL=m

NOTE to 5.: CONFIG_CRYPTO_CRC32C_INTEL=m

NOTE to 6.: Required for some kernel-modules like out-of-tree VirtualBox or CONFIG_LKDTM=m (see #1120).

Update 16-Aug-2020:
Replaced patches both DWARF patches by...
Introduce CONFIG_DEBUG_INFO_AS_DWARF2 and CONFIG_DEBUG_INFO_AS_DWARF4
Link: https://patchwork.kernel.org/patch/11716107/

Update 19-Aug-2020:
All x86/crypto patches now in Linux v5.8.2 (and other Linux-stable releases)

Update 05-Aug-2020:
All x86/crypto patches now in Linux 5.9

Update 03-Aug-2020:
Linux version 5.8 boots fine, see #1117 (comment)

Update 01-Aug-2020:
Add @arndb x86: work around clang IAS bug referencing __force_order patch to be able to build VirtualBox out-of-tree kernel-module (see #1104 (comment))

Update 23-Jul-2020:
All x86/crypto patches now in https://git.kernel.org/pub/scm/linux/kernel/git/herbert/cryptodev-2.6.git/.

Update 22-Jul-2020:
Initial version

dileks · 2020-08-18T17:58:36Z

Update: Linux v5.9-rc1:

$ cat /proc/version 
Linux version 5.9.0-rc1-3-amd64-llvm11-ias (sedat.dilek@gmail.com@iniza) (clang version 11.0.0 (https://github.com/llvm/llvm-project c2f52e2c1288b00eae528825fb92668b1f3df732), LLD 11.0.0 (https://github.com/llvm/llvm-project c2f52e2c1288b00eae528825fb92668b1f3df732)) #3~bullseye+dileks1 SMP 2020-08-18

Required patches/patchsets:

x86: work around clang IAS bug referencing __force_order
Link: https://lore.kernel.org/patchwork/patch/1248382/
Fixes: Building VirtualBox kernel modules with Clang and DKMS #1104 (comment)
Fixes: ERROR: modpost: "__force_order" [drivers/misc/lkdtm/lkdtm.ko] undefined! #1120
x86/boot: Remove run-time relocations from compressed kernel v6
Link: https://lore.kernel.org/patchwork/project/lkml/list/?series=456251
Git: https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/log/?h=x86/boot
Fixes: ld.lld-11: error: section ... file range overlaps with ... #1129

Optional DWARF version 4:

Introduce CONFIG_DEBUG_INFO_AS_DWARF2 and CONFIG_DEBUG_INFO_AS_DWARF4 (see https://patchwork.kernel.org/patch/11716107/)
kbuild: Simplify DEBUG_INFO Kconfig handling (see https://patchwork.kernel.org/patch/11716109/)

Perf stat:

 Performance counter stats for 'make V=1 -j3 HOSTCC=clang-11 HOSTCXX=clang++-11 HOSTLD=ld.lld-11 HOSTAR=llvm-ar-11 CC=clang-11 LD=ld.lld-11 AR=llvm-ar-11 NM=llvm-nm-11 OBJCOPY=llvm-objcopy-11 OBJDUMP=llvm-objdump-11 OBJSIZE=llvm-size-11 READELF=llvm-readelf-11 STRIP=llvm-strip-11 LLVM_IAS=1 LOCALVERSION=-3-amd64-llvm11-ias KBUILD_VERBOSE=1 KBUILD_BUILD_HOST=iniza KBUILD_BUILD_USER=sedat.dilek@gmail.com KBUILD_BUILD_TIMESTAMP=2020-08-18 bindeb-pkg KDEB_PKGVERSION=5.9.0~rc1-3~bullseye+dileks1':

       41454490.82 msec task-clock:u              #    2.896 CPUs utilized          
                 0      context-switches:u        #    0.000 K/sec                  
                 0      cpu-migrations:u          #    0.000 K/sec                  
         308569350      page-faults:u             #    0.007 M/sec                  
    75665754549051      cycles:u                  #    1.825 GHz                    
    60254260110478      stalled-cycles-frontend:u #   79.63% frontend cycles idle   
    48829161160508      stalled-cycles-backend:u  #   64.53% backend cycles idle    
    48186232262620      instructions:u            #    0.64  insn per cycle         
                                                  #    1.25  stalled cycles per insn
     9222134517271      branches:u                #  222.464 M/sec                  
      389598075528      branch-misses:u           #    4.22% of all branches        

   14311.995383874 seconds time elapsed

   39452.053992000 seconds user
    2058.547318000 seconds sys

Attachments: My kernel-config and dmesg-output

config-5.9.0-rc1-3-amd64-llvm11-ias.txt
dmesg-T_5.9.0-rc1-3-amd64-llvm11-ias.txt

nickdesaulniers · 2020-09-15T23:34:20Z

please file individual bug reports for specific failures specific to either a missing feature or particular translation unit.

dileks · 2020-09-16T07:14:53Z

@nickdesaulniers

Clang-IAS is fine with LLVM toolchain version 11 rc2+ and recent Linux-kernel version like 5.8 (with some upstream fixes) or 5.9-rcX.

Currently, I use Linux version 5.9-rc5 and LLVM toolchain version 11.0.0 (https://github.com/llvm/llvm-project 97ac9e82002d6b12831ca2c78f739cca65a4fa05) (ThinLTO and PGO optimized) with no issues.

What is the status with LLVM toolchain version 10.0.1 (which will be default with Linux version 5.10)?
Which Linux-kernel version supports fully Clang-IAS?
Do we have all Clang-IAS related patches in all backports (Linux and LLVM toolchain)?

As you better know Clang-IAS is prerequisite for Clang-LTO and Clang-CFI.

I left this issue open that people have an orientation.
If you have a better solution or maybe there is a link I do not know off?

nickdesaulniers · 2020-09-16T19:03:23Z

Which Linux-kernel version supports fully Clang-IAS?

None. LLVM_IAS=1 is experimental/WIP for now.

If you have a better solution or maybe there is a link I do not know off?

Either the label "integrated-as" or maybe a github "milestone" might be more appropriate?

dileks added the question Question asked by issue author. label Jun 11, 2020

nickdesaulniers mentioned this issue Jun 11, 2020

arch/x86/crypto/aesni-intel_asm.S broken with LLVM_IAS=1 #1050

Closed

nickdesaulniers closed this as completed Jun 11, 2020

dileks reopened this Jun 12, 2020

dileks mentioned this issue Jun 16, 2020

error: invalid reassignment of non-absolute variable 'var_ddq_add' #1008

Closed

dileks added the [ARCH] x86_64 This bug impacts ARCH=x86_64 label Jul 10, 2020

torvic9 mentioned this issue Jul 22, 2020

Building VirtualBox kernel modules with Clang and DKMS #1104

Closed

This was referenced Aug 3, 2020

Linux version 5.8 released #1117

Closed

[PATCH] x86/crypto: aesni: Fix build with LLVM_IAS=1 (next part for 5.9+) #1118

Closed

dileks removed the question Question asked by issue author. label Aug 5, 2020

dileks changed the title ~~Experimenting with integrated assembler aka LLVM_IAS=1~~ Status: Building with Clang's integrated assembler aka LLVM_IAS=1 Aug 18, 2020

nickdesaulniers closed this as completed Sep 15, 2020

Status: Building with Clang's integrated assembler aka LLVM_IAS=1 #1049

Status: Building with Clang's integrated assembler aka LLVM_IAS=1 #1049

Comments

dileks commented Jun 11, 2020

dileks commented Jun 11, 2020

dileks commented Jun 11, 2020

dileks commented Jun 11, 2020

dileks commented Jun 11, 2020

dileks commented Jun 11, 2020

dileks commented Jun 11, 2020

dileks commented Jun 11, 2020 • edited

dileks commented Jun 11, 2020 • edited

dileks commented Jun 11, 2020

dileks commented Jun 11, 2020

dileks commented Jun 11, 2020 • edited

nickdesaulniers commented Jun 11, 2020

dileks commented Jun 11, 2020

dileks commented Jun 11, 2020

dileks commented Jun 11, 2020 • edited

nickdesaulniers commented Jun 12, 2020

dileks commented Jun 12, 2020

dileks commented Jun 12, 2020 • edited by nathanchance

dileks commented Jun 12, 2020 • edited

dileks commented Jun 12, 2020 • edited

E5ten commented Jun 13, 2020

dileks commented Jun 14, 2020 • edited

dileks commented Jun 14, 2020

dileks commented Jun 14, 2020

dileks commented Jun 14, 2020 • edited

dileks commented Jun 14, 2020 • edited

dileks commented Jun 15, 2020

E5ten commented Jun 15, 2020

dileks commented Jun 15, 2020

dileks commented Jun 15, 2020 • edited

dileks commented Jun 15, 2020

dileks commented Jun 15, 2020

dileks commented Jun 16, 2020 • edited

dileks commented Jun 16, 2020

dileks commented Jun 17, 2020

dileks commented Jun 17, 2020 • edited

dileks commented Jun 17, 2020

dileks commented Jun 22, 2020

dileks commented Jun 23, 2020

dileks commented Jun 23, 2020

dileks commented Jun 30, 2020

dileks commented Jul 10, 2020 • edited by nathanchance

dileks commented Jul 12, 2020

dileks commented Jul 22, 2020 • edited

dileks commented Aug 18, 2020 • edited

nickdesaulniers commented Sep 15, 2020

dileks commented Sep 16, 2020

nickdesaulniers commented Sep 16, 2020

dileks commented Jun 11, 2020 •

edited

dileks commented Jun 11, 2020 •

edited

dileks commented Jun 11, 2020 •

edited

dileks commented Jun 11, 2020 •

edited

dileks commented Jun 12, 2020 •

edited by nathanchance

dileks commented Jun 12, 2020 •

edited

dileks commented Jun 12, 2020 •

edited

dileks commented Jun 14, 2020 •

edited

dileks commented Jun 14, 2020 •

edited

dileks commented Jun 14, 2020 •

edited

dileks commented Jun 15, 2020 •

edited

dileks commented Jun 16, 2020 •

edited

dileks commented Jun 17, 2020 •

edited

dileks commented Jul 10, 2020 •

edited by nathanchance

dileks commented Jul 22, 2020 •

edited

dileks commented Aug 18, 2020 •

edited