llvm-objcopy produces corrupted .debug_str for elf32-x86-64 (X32 ABI) emulation (Z_DATA_ERROR)

[kiroma]

ld.lld: error: arch/x86/entry/vdso/vgetcpu-x32.o:(.debug_str): uncompress failed: zlib error: Z_DATA_ERROR
Tried compiling 5.2-rc4

Clang compiled from the git monorepo, two stage build
clang version 9.0.0 (https://github.com/llvm/llvm-project.git f7ba8b808a89d5e050deb7dbd77004fa8e0dff9b)

[nathanchance]

What is your .config?

[kiroma]

Copied over from OpenSUSE's default config: https://gist.github.com/kiroma/f7766f62f50974db360ac9ffc20ac364
Ran menuconfig once to account for compiler change.

[nathanchance]

I ran:

$ mkdir -p out
$ curl -LSso out/.config https://gist.github.com/kiroma/f7766f62f50974db360ac9ffc20ac364/raw/ea9253d4becc98da16ad7cf4501fb00025d101c8/.config
$ make -j$(nproc) CC=clang LD=ld.lld O=out olddefconfig modules_prepare bzImage modules

and I don't see this error (I see a couple of other ones, namely #327 and ERROR: "__memcat_p" [drivers/hwtracing/stm/stm_core.ko] undefined!, which I'll investigate and open a separate issue for).

What were your LLVM config build options?

[kiroma]

I ran:

make -j$(nproc) CC=clang AS=clang LD=ld.lld AR=llvm-ar NM=llvm-nm STRIP=llvm-strip OBJCOPY=llvm-objcopy OBJDUMP=llvm-objdump HOSTCC=clang HOSTLD=ld.lld HOSTAR=llvm-ar

I configured the build with
-DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;libunwind;libcxx;libcxxabi;compiler-rt;lld;polly" -DCLANG_DEFAULT_LINKER=lld -DLLVM_ENABLE_LTO=Thin -DLLVM_TARGETS_TO_BUILD=host -DCMAKE_BUILD_TYPE=Release
The first stage was compiled using gcc-9 without LTO.

[nathanchance]

I am going to build with your options to see if I can reproduce it.

In the meantime, would you mind trying out our build script here to see if it still happens there?

[nathanchance]

Alright, this is reproducible using my LLVM toolchain compiled using tc-build and it is related to llvm-objcopy.

$ git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/
$ cd linux
$ git checkout v5.2-rc4
$ mkdir -p out
$ curl -LSso out/.config https://gist.github.com/kiroma/f7766f62f50974db360ac9ffc20ac364/raw/ea9253d4becc98da16ad7cf4501fb00025d101c8/.config

No error:

$ make -j$(nproc) CC=clang LD=ld.lld O=out olddefconfig arch/x86/entry/vdso/

Error:

$ make -j$(nproc) CC=clang LD=ld.lld O=out OBJCOPY=llvm-objcopy olddefconfig arch/x86/entry/vdso/

It looks like the llvm-objcopy call for vgetcpu-x32.o is llvm-objcopy -O elf32-x86-64 arch/x86/entry/vdso/vgetcpu.o arch/x86/entry/vdso/vgetcpu-x32.o according to V=1.

cc @rupprecht

[MaskRay]

Cannot reproduce with https://gist.github.com/kiroma/f7766f62f50974db360ac9ffc20ac364/raw/ea9253d4becc98da16ad7cf4501fb00025d101c8/.config, master or v5.2-rc4: make -j$(nproc) CC=clang LD=ld.lld O=out OBJCOPY=llvm-objcopy olddefconfig arch/x86/entry/vdso/

Can you check your llvm CMakeCache.txt have the following lines?

HAVE_ZLIB_H:INTERNAL=1
LLVM_ENABLE_ZLIB:BOON=ON

[nathanchance]

Yes:

$ fd CMakeCache.txt /mnt/build/llvm -x rg -H -n "HAVE_ZLIB_H|LLVM_ENABLE_ZLIB"
/mnt/build/llvm/stage2/CMakeCache.txt:786:LLVM_ENABLE_ZLIB:BOOL=ON
/mnt/build/llvm/stage2/CMakeCache.txt:1774:HAVE_ZLIB_H:INTERNAL=1
/mnt/build/llvm/stage3/CMakeCache.txt:856:LLVM_ENABLE_ZLIB:BOOL=ON
/mnt/build/llvm/stage3/CMakeCache.txt:2061:HAVE_ZLIB_H:INTERNAL=1
/mnt/build/llvm/stage1/CMakeCache.txt:850:LLVM_ENABLE_ZLIB:BOOL=ON
/mnt/build/llvm/stage1/CMakeCache.txt:2055:HAVE_ZLIB_H:INTERNAL=1

Was that with the .config linked? I can still reproduce this on v5.2-rc6 as of r364149. Could this be a zlib host issue?

[nickdesaulniers]

I was not able to reproduce.

$ wget https://gist.githubusercontent.com/kiroma/f7766f62f50974db360ac9ffc20ac364/raw/ea9253d4becc98da16ad7cf4501fb00025d101c8/.config -O report.config
$ cp report.config .config
$ make CC=clang LD=ld.lld OBJCOPY=llvm-objcopy -j71 olddefconfig
$ make CC=clang LD=ld.lld OBJCOPY=llvm-objcopy -j71 arch/x86/entry/vdso/

@nathanchance @kiroma , if you can still reproduce, please attach:

1. the llvm-objcopy invocation from adding V=1 to your make invocation.
2. the relevant .o files from the command so that others can inspect them.

There are two llvm commits that mentions ZLIB and llvm-objcopy around the time of the report is:

1. llvm/llvm-project@f2eb8ca.
2. llvm/llvm-project@ade3c70
   Though they seem related to the use of --compress-debug-sections which I don't see in the kernel sources. Both of them landed in llvmorg-9.0.0-rc1.

[nathanchance]

So this is interesting.

$ curl -LSso .config https://gist.github.com/kiroma/f7766f62f50974db360ac9ffc20ac364/raw/ea9253d4becc98da16ad7cf4501fb00025d101c8/.config

$ make -j$(nproc) CC=clang LD=ld.lld OBJCOPY=llvm-objcopy olddefconfig arch/x86/entry/vdso/
...
  CC      arch/x86/entry/vdso/vclock_gettime.o
  LDS     arch/x86/entry/vdso/vdso32/vdso32.lds
  CC      arch/x86/entry/vdso/vgetcpu.o
  AS      arch/x86/entry/vdso/vdso32/note.o
  CC      arch/x86/entry/vdso/vdso32/vclock_gettime.o
  AS      arch/x86/entry/vdso/vdso32/system_call.o
  AS      arch/x86/entry/vdso/vdso32/sigreturn.o
  X32     arch/x86/entry/vdso/vdso-note-x32.o
  X32     arch/x86/entry/vdso/vgetcpu-x32.o
  VDSO    arch/x86/entry/vdso/vdso64.so.dbg
  X32     arch/x86/entry/vdso/vclock_gettime-x32.o
  VDSO    arch/x86/entry/vdso/vdsox32.so.dbg
  VDSO    arch/x86/entry/vdso/vdso32.so.dbg
  OBJCOPY arch/x86/entry/vdso/vdso64.so
  OBJCOPY arch/x86/entry/vdso/vdsox32.so
  OBJCOPY arch/x86/entry/vdso/vdso32.so
  VDSO2C  arch/x86/entry/vdso/vdso-image-64.c
  VDSO2C  arch/x86/entry/vdso/vdso-image-x32.c
  VDSO2C  arch/x86/entry/vdso/vdso-image-32.c
  CC      arch/x86/entry/vdso/vdso-image-64.o
  CC      arch/x86/entry/vdso/vdso-image-x32.o
  CC      arch/x86/entry/vdso/vdso-image-32.o
  AR      arch/x86/entry/vdso/built-in.a

works when I am using my host's binutils (Ubuntu 18.04, binutils 2.30). As soon as I switch over to binutils 2.34, built using build-binutils.py, it breaks.

$ curl -LSso .config https://gist.github.com/kiroma/f7766f62f50974db360ac9ffc20ac364/raw/ea9253d4becc98da16ad7cf4501fb00025d101c8/.config

$ PATH=${HOME}/cbl/git/tc-build/install/bin:${PATH} make -j$(nproc) CC=clang LD=ld.lld OBJCOPY=llvm-objcopy olddefconfig arch/x86/entry/vdso/
...
  X32     arch/x86/entry/vdso/vclock_gettime-x32.o
  VDSO    arch/x86/entry/vdso/vdsox32.so.dbg
  OBJCOPY arch/x86/entry/vdso/vdso64.so
ld.lld: error: arch/x86/entry/vdso/vgetcpu-x32.o:(.debug_str): uncompress failed: zlib error: Z_DATA_ERROR
ld.lld: error: arch/x86/entry/vdso/vclock_gettime-x32.o:(.debug_str): uncompress failed: zlib error: Z_DATA_ERROR
objdump: 'arch/x86/entry/vdso/vdsox32.so.dbg': No such file
  VDSO    arch/x86/entry/vdso/vdso32.so.dbg
  VDSO2C  arch/x86/entry/vdso/vdso-image-64.c
  OBJCOPY arch/x86/entry/vdso/vdsox32.so
  OBJCOPY arch/x86/entry/vdso/vdso32.so
llvm-objcopy: error: 'arch/x86/entry/vdso/vdsox32.so.dbg': No such file or directory
make[4]: *** [arch/x86/entry/vdso/Makefile:125: arch/x86/entry/vdso/vdsox32.so] Error 1
...

[nickdesaulniers]

Time to bisect binutils. ;)

[nathanchance]

Building binutils-2_30 with or without --with-system-zlib results in the same issue so something else is going on...

[nickdesaulniers]

Though they seem related to the use of --compress-debug-sections which I don't see in the kernel sources.

ld.lld: error: arch/x86/entry/vdso/vgetcpu-x32.o:(.debug_str): uncompress failed: zlib error: Z_DATA_ERROR

.debug_str and zlib sounds a lot like "compressed debug sections" to me. Maybe someone (bintutils) has a default setting?

if you can still reproduce, please attach:
the relevant .o files from the command so that others can inspect them.

🌵 🌮

[nathanchance]

.debug_str and zlib sounds a lot like "compressed debug sections" to me. Maybe someone (bintutils) has a default setting?

Possible...

the relevant .o files from the command so that others can inspect them.

Yup, sorry!

cbl-514-out.x86_64.tar.gz

[nickdesaulniers]

$ gunzip cbl-514-out.x86_64.tar.gz
$ tar xf cbl-514-out.x86_64.tar
$ ld.lld arch/x86/entry/vdso/vgetcpu-x32.o                 
ld.lld: error: arch/x86/entry/vdso/vgetcpu-x32.o:(.debug_str): uncompress failed: zlib error: Z_DATA_ERROR
$ llvm-readobj -sections arch/x86/entry/vdso/vgetcpu-x32.o
...
  Section {
    Index: 11
    Name: .debug_str (79)
    Type: SHT_PROGBITS (0x1)
    Flags [ (0x830)
      SHF_COMPRESSED (0x800)
      SHF_MERGE (0x10)
      SHF_STRINGS (0x20)
    ]
    Address: 0x0
    Offset: 0x226
    Size: 266
    Link: 0
    Info: 0
    AddressAlignment: 1
    EntrySize: 1
  }
...

I can slice it out with: dd if=arch/x86/entry/vdso/vgetcpu-x32.o of=foo.o skip=550 bs=1 count=220 but not really sure how to verify it.

Next command we need is make V=1 for arch/x86/entry/vdso/vgetcpu-x32.o to see how it was built. Maybe -no-integrates-as invoked gas and gas got it wrong? Otherwise clang's object streamer did.

[nickdesaulniers]

$ llvm-dwarfdump arch/x86/entry/vdso/vgetcpu-x32.o
error: failed to decompress '.debug_loc', zlib error: Z_DATA_ERROR
error: failed to decompress '.debug_abbrev', zlib error: Z_DATA_ERROR
error: failed to decompress '.debug_info', zlib error: Z_DATA_ERROR
error: failed to decompress '.debug_str', zlib error: Z_DATA_ERROR
error: failed to decompress '.debug_line', zlib error: Z_DATA_ERROR
...

[nickdesaulniers]

Next command we need is make V=1 for arch/x86/entry/vdso/vgetcpu-x32.o to see how it was built.

Looks like it's just:

$ llvm-objcopy -O elf32-x86-64 arch/x86/entry/vdso/vgetcpu.o arch/x86/entry/vdso/vgetcpu-x32.o

in @nathanchance 's repo, llvm-dwarfdump is happy with arch/x86/entry/vdso/vgetcpu.o, but not arch/x86/entry/vdso/vgetcpu-x32.o. Did llvm-objcopy mess this up?

[nickdesaulniers]

Ok, I think I can reliably reproduce the bug, via @nathanchance 's provided arch/x86/entry/vdso/vgetcpu.o:

$ llvm-dwarfdump arch/x86/entry/vdso/vgetcpu.o &>/dev/null
$ echo $?
0
$ llvm-objcopy -O elf32-x86-64 arch/x86/entry/vdso/vgetcpu.o foo.o
$ llvm-dwarfdump foo.o 
error: failed to decompress '.debug_loc', zlib error: Z_DATA_ERROR
error: failed to decompress '.debug_abbrev', zlib error: Z_DATA_ERROR
error: failed to decompress '.debug_info', zlib error: Z_DATA_ERROR
error: failed to decompress '.debug_str', zlib error: Z_DATA_ERROR
error: failed to decompress '.debug_line', zlib error: Z_DATA_ERROR

[nickdesaulniers]

also, this looks related to x32, but I don't see CONFIG_X86_X32_ABI=y set in the provided config...oh, maybe CONFIG_X86_X32 ("x32 ABI for 64-bit mode")

2900     Include code to run binaries for the x32 native 32-bit ABI                                                                  
2901     for 64-bit processors.  An x32 process gets access to the                                                                   
2902     full 64-bit register file and wide data path while leaving                                                                  
2903     pointers at 32 bits for smaller memory footprint.                                                                           
2904                                                                                                                                 
2905     You will need a recent binutils (2.22 or later) with                                                                        
2906     elf32_x86_64 support enabled to compile a kernel with this                                                                  
2907     option set.

yikes.

Ah CONFIG_X86_X32_ABI gets set in arch/x86/Makefile:

136 ifdef CONFIG_X86_X32                                                                                                             
137   x32_ld_ok := $(call try-run,\                                                                                                  
138       /bin/echo -e '1: .quad 1b' | \                                                                                             
139       $(CC) $(KBUILD_AFLAGS) -c -x assembler -o "$$TMP" - && \                                                                   
140       $(OBJCOPY) -O elf32-x86-64 "$$TMP" "$$TMPO" && \                                                                           
141       $(LD) -m elf32_x86_64 "$$TMPO" -o "$$TMP",y,n)                                                                             
142         ifeq ($(x32_ld_ok),y)                                                                                                    
143                 CONFIG_X86_X32_ABI := y                                                                                          
144                 KBUILD_AFLAGS += -DCONFIG_X86_X32_ABI                                                                            
145                 KBUILD_CFLAGS += -DCONFIG_X86_X32_ABI                                                                            
146         else                                                                                                                     
147                 $(warning CONFIG_X86_X32 enabled but no binutils support)                                                        
148         endif                                                                                                                    
149 endif                                                                                                                            
150 export CONFIG_X86_X32_ABI