[RISCV] LLVM produces binaries where GNU objdump is unable to use the mapping symbols for proper disassembly

[asb]

It's possible there's a GNU bug here, but it's the case that clang produces binaries where GNU objdump can't use the mapping symbols to properly print some instructions, while llvm-objdump manages fine. But binaries produced by gcc are disassembled properly with either objdump implementation.

Consider the following example, where max is a trivial function that will result in the selection of max from the Zbb extension.

cat - <<EOF > max.c
int max(int x, int y) {
  return x > y ? x : y;
}

int main(void) {
  return 0;
}
EOF
riscv64-linux-gnu-gcc -c -O3 -march=rv64gc_zbb max.c -o max.gcc.o
$HOME/llvm-project/build/default/bin/clang --target=riscv64-linux-gnu -c -O3 -march=rv64gc_zbb max.c -o max.clang.o
riscv64-linux-gnu-gcc -O3 -march=rv64gc_zbb max.c -o max.gcc
$HOME/llvm-project/build/default/bin/clang --target=riscv64-linux-gnu -O3 -march=rv64gc_zbb --sysroot=$HOME/rvsysroot -fuse-ld=lld max.c -o max.clang
riscv64-linux-gnu-gcc -O3 -fuse-ld=lld -march=rv64gc_zbb max.c -o max.gcc.lld

The result is:

• max.gcc.o: Disassembles fine for LLVM and GNU objdump (i.e. max a0, a1, a0 is printed)
• max.clang.o: Disassembles fine for LLVM and GNU objdump
• max.gcc: Disassembles fine for LLVM and GNU objdump
• max.gcc.lld: Disassembles fine for LLVM and GNU objdump
• max.clang: Disassembles fine for LLVM objdump, but for GNU objdump we get .insn printed for the max instruction
  • 169a: 0ab56533 .insn 4, 0x0ab56533

I see this across basically all LLVM/Clang produced binaries.

CC @kito-cheng in case you've seen this as well or have any insight?

[llvmbot]

@llvm/issue-subscribers-backend-risc-v

Author: Alex Bradbury (asb)

It's possible there's a GNU bug here, but it's the case that `clang` produces binaries where GNU objdump can't use the mapping symbols to properly print some instructions, while `llvm-objdump` manages fine. But binaries produced by `gcc` are disassembled properly with either objdump implementation.

Consider the following example, where max is a trivial function that will result in the selection of max from the Zbb extension.

cat - <<EOF > max.c
int max(int x, int y) {
  return x > y ? x : y;
}

int main(void) {
  return 0;
}
EOF
riscv64-linux-gnu-gcc -c -O3 -march=rv64gc_zbb max.c -o max.gcc.o
$HOME/llvm-project/build/default/bin/clang --target=riscv64-linux-gnu -c -O3 -march=rv64gc_zbb max.c -o max.clang.o
riscv64-linux-gnu-gcc -O3 -march=rv64gc_zbb max.c -o max.gcc
$HOME/llvm-project/build/default/bin/clang --target=riscv64-linux-gnu -O3 -march=rv64gc_zbb --sysroot=$HOME/rvsysroot -fuse-ld=lld max.c -o max.clang
riscv64-linux-gnu-gcc -O3 -fuse-ld=lld -march=rv64gc_zbb max.c -o max.gcc.lld

The result is:

• max.gcc.o: Disassembles fine for LLVM and GNU objdump
• max.clang.o: Disassembles fine for LLVM and GNU objdump
• max.gcc: Disassembles fine for LLVM and GNU objdump
• max.gcc.lld: Disassembles fine for LLVM and GNU objdump
• max.clang: Disassembles fine for LLVM objdump, but for GNU objdump we get .insn printed for the max instruction
  • 169a: 0ab56533 .insn 4, 0x0ab56533

I see this across basically all LLVM/Clang produced binaries.

CC @kito-cheng in case you've seen this as well or have any insight?

[palmer-dabbelt]

I didn't look closely, but maybe it's this https://inbox.sourceware.org/binutils/CAPpQWtBTaoQ1fnvVKR6Vdr=ayRoemAOtN2_y32Bt0e+7NZvNaQ@mail.gmail.com/T/#t?

[Nelson1225]

Yeah that should be the same issue.

GNU made an implementation very early to provide a reference for the psABI specification before, and at that time we defined $x to be the same as the isa of the previous $x+isa, and forced to generate $x+isa at first for all sections. The reason for this was because, considering two objects with different isa are linked,

% cat a.s
.text
addi a0, a1, 0
.word 0
nop
% cat b.s
.text
addi a0, a1, 1
.word 1
nop
% riscv64-unknown-elf-as -march=rv64ic a.s -o a.o
% riscv64-unknown-elf-as -march=rv64ia b.s -o b.o
% riscv64-unknown-elf-ld a.o b.o -o c
% riscv64-unknown-elf-readelf -Ws c  

Symbol table '.symtab' contains 19 entries:
   Num:    Value          Size Type    Bind   Vis      Ndx Name
...
     3: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS a.o
     4: 00000000000100b0     0 NOTYPE  LOCAL  DEFAULT    1 $xrv64i2p1_c2p0
     5: 00000000000100b2     0 NOTYPE  LOCAL  DEFAULT    1 $d
     6: 00000000000100b6     0 NOTYPE  LOCAL  DEFAULT    1 $x
     7: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS b.o
     8: 00000000000100b8     0 NOTYPE  LOCAL  DEFAULT    1 $xrv64i2p1_a2p1_zaamo1p0_zalrsc1p0
     9: 00000000000100bc     0 NOTYPE  LOCAL  DEFAULT    1 $d
    10: 00000000000100c0     0 NOTYPE  LOCAL  DEFAULT    1 $x
...
%riscv64-unknown-elf-readelf -A c 
Attribute Section: riscv
File Attributes
  Tag_RISCV_arch: "rv64i2p1_m2p0_a2p1_f2p2_d2p2_zicsr2p0_zifencei2p0_zmmul1p0_zaamo1p0_zalrsc1p0"

So linker won't do anything when merging two objects with different isa. According to the current spec definition, $x is same as the elf arch attribute, so the symbol table may be like the follows if linker doesn't do anything,

     3: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS a.o
     4: 00000000000100b0     0 NOTYPE  LOCAL  DEFAULT    1 $x
     5: 00000000000100b2     0 NOTYPE  LOCAL  DEFAULT    1 $d
     6: 00000000000100b6     0 NOTYPE  LOCAL  DEFAULT    1 $x
     7: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS b.o
     8: 00000000000100b8     0 NOTYPE  LOCAL  DEFAULT    1 $x
     9: 00000000000100bc     0 NOTYPE  LOCAL  DEFAULT    1 $d
    10: 00000000000100c0     0 NOTYPE  LOCAL  DEFAULT    1 $x

For the above case, linker will need to re-write all $x to corresponding $x+isa, if linked objects with different isa, which seems no good from my side. For shared lib, maybe linker should always re-write $x since we never know what arch objects it will work with. @palmer-dabbelt and I were discussing this for many days, for GNU side, we will stop to generate $x by assembler, and let objdump recognizes $x as the same as elf arch attribute, after 2.45 release until there will be a better solution in the psABI. We already landed the implementation and made a reference very early, but unfortunately psABI chose another definition, so the best we can do is to stop using any $x for now.

Thanks
Nelson

[asb]

Patches for GNU objdump have been posted (thanks Nelson!) https://sourceware.org/pipermail/binutils/2025-February/139625.html

I'll raise this in the RISC-V sync-up call today in case anyone feels compelled to do something other than wait for the fix to be shipped (e.g. it sounds like we could change LLVM to emit explicit mapping symbols at the start of each section).