llvm-objcopy missing padding between sections

[Kimplul]

I have an elf file that I would like to turn into a raw binary, and I'm trying to allocate the .bss section directly in the binary. For some reason, there is a small gap between my .text and .bss, which is padded with some bytes when using GNU objcopy, but seemingly ignored when using llvm-objcopy.

To illustrate:
riscv64-unknown-elf-objcopy -Obinary --set-section-flags .bss=alloc,load,contents --gap-fill 255 example.elf example.bin

produces a raw binary ending in

...
00000240: ef00 e000 0100 e270 4274 2161 8280 2a86  .......pBt!a..*.
00000250: 2e85 0286 ffff ffff 0000 0000 0000 0000  ................

where ffff ffff is padding. (The actual value of the padding is irrelevant, I just used ff for demonstration purposes). The equivalent command with llvm-objcopy
llvm-objcopy -Obinary --set-section-flags .bss=alloc,load,contents example.elf example.bin

produces

...
00000240: ef00 e000 0100 e270 4274 2161 8280 2a86  .......pBt!a..*.
00000250: 2e85 0286 0000 0000 0000 0000            ............

For context, here's the readelf output for example.elf:

readelf -SW example.elf
There are 15 section headers, starting at offset 0x20d0:

Section Headers:
  [Nr] Name              Type            Address          Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            0000000000000000 000000 000000 00      0   0  0
  [ 1] .text             PROGBITS        0000000080080000 001000 000254 00  AX  0   0  2
  [ 2] .bss              NOBITS          0000000080080258 001254 000008 00  WA  0   0  8
  [ 3] .debug_info       PROGBITS        0000000000000000 001254 00030e 00      0   0  1
  [ 4] .debug_abbrev     PROGBITS        0000000000000000 001562 000187 00      0   0  1
  [ 5] .debug_aranges    PROGBITS        0000000000000000 0016f0 000060 00      0   0 16
  [ 6] .debug_line       PROGBITS        0000000000000000 001750 000292 00      0   0  1
  [ 7] .debug_str        PROGBITS        0000000000000000 0019e2 0001aa 01  MS  0   0  1
  [ 8] .debug_line_str   PROGBITS        0000000000000000 001b8c 000098 01  MS  0   0  1
  [ 9] .comment          PROGBITS        0000000000000000 001c24 000012 01  MS  0   0  1
  [10] .riscv.attributes RISCV_ATTRIBUTES 0000000000000000 001c36 000039 00      0   0  1
  [11] .debug_frame      PROGBITS        0000000000000000 001c70 0000d0 00      0   0  8
  [12] .symtab           SYMTAB          0000000000000000 001d40 000270 18     13  17  8
  [13] .strtab           STRTAB          0000000000000000 001fb0 000080 00      0   0  1
  [14] .shstrtab         STRTAB          0000000000000000 002030 00009e 00      0   0  1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  D (mbind), p (processor specific)

I'm assuming the bug is on LLVM's side in this case, because the start address of .bss matches when using GNU objcopy, whereas its four bytes short when using LLVM.

[llvmbot]

@llvm/issue-subscribers-tools-llvm-objcopy-strip

[jh7370]

@Kimplul, I agree that things don't look quite right - it looks as though llvm-objcopy isn't writing alignment padding to me. Could you provide a dump of the program headers, please? Also, what happens if you write to an ELF instead of raw binary?

[Kimplul]

Program headers of initial elf file:

readelf -lW example.elf

Elf file type is EXEC (Executable file)
Entry point 0x80080000
There are 2 program headers, starting at offset 64

Program Headers:
  Type           Offset   VirtAddr           PhysAddr           FileSiz  MemSiz   Flg Align
  RISCV_ATTRIBUT 0x001c36 0x0000000000000000 0x0000000000000000 0x000039 0x000000 R   0x1
  LOAD           0x001000 0x0000000080080000 0x0000000000000000 0x000254 0x000260 RWE 0x1000

 Section to Segment mapping:
  Segment Sections...
   00     .riscv.attributes
   01     .text .bss

When creating an ELF file, there seems to still be some differences between GNU and LLVM obcjopy.

LLVM:

llvm-objcopy -Oelf64-littleriscv --set-section-flags .bss=alloc,load,contents example.elf llvmexample.elf
readelf -aW llvmexample.elf
...
Section Headers:
  [Nr] Name              Type            Address          Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            0000000000000000 000000 000000 00      0   0  0
  [ 1] .text             PROGBITS        0000000080080000 001000 000254 00  AX  0   0  2
  [ 2] .bss              PROGBITS        0000000080080258 001254 000008 00  WA  0   0  8
  [ 3] .debug_info       PROGBITS        0000000000000000 00128d 00030e 00      0   0  1
  [ 4] .debug_abbrev     PROGBITS        0000000000000000 00159b 000187 00      0   0  1
  [ 5] .debug_aranges    PROGBITS        0000000000000000 001730 000060 00      0   0 16
  [ 6] .debug_line       PROGBITS        0000000000000000 001790 000292 00      0   0  1
  [ 7] .debug_str        PROGBITS        0000000000000000 001a22 0001aa 01  MS  0   0  1
  [ 8] .debug_line_str   PROGBITS        0000000000000000 001bcc 000098 01  MS  0   0  1
  [ 9] .comment          PROGBITS        0000000000000000 001c64 000012 01  MS  0   0  1
  [10] .riscv.attributes RISCV_ATTRIBUTES 0000000000000000 001254 000039 00      0   0  1
  [11] .debug_frame      PROGBITS        0000000000000000 001c78 0000d0 00      0   0  8
  [12] .symtab           SYMTAB          0000000000000000 001d48 000270 18     13  17  8
  [13] .strtab           STRTAB          0000000000000000 001fb8 000080 00      0   0  1
  [14] .shstrtab         STRTAB          0000000000000000 002038 00009e 00      0   0  1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  D (mbind), p (processor specific)

There are no section groups in this file.

Program Headers:
  Type           Offset   VirtAddr           PhysAddr           FileSiz  MemSiz   Flg Align
  RISCV_ATTRIBUT 0x001254 0x0000000000000000 0x0000000000000000 0x000039 0x000000 R   0x1
  LOAD           0x001000 0x0000000080080000 0x0000000000000000 0x000254 0x000260 RWE 0x1000

 Section to Segment mapping:
  Segment Sections...
   00     .riscv.attributes
   01     .text
...

GNU:

riscv64-unknown-elf-objcopy -Oelf64-littleriscv --set-section-flags .bss=alloc,load,contents example.elf gnuexample.elf
readelf -aW gnuexample.elf
...
Section Headers:
  [Nr] Name              Type            Address          Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            0000000000000000 000000 000000 00      0   0  0
  [ 1] .text             PROGBITS        0000000080080000 001000 000254 00  AX  0   0  2
  [ 2] .bss              PROGBITS        0000000080080258 001258 000008 00  WA  0   0  8
  [ 3] .debug_info       PROGBITS        0000000000000000 001260 00030e 00      0   0  1
  [ 4] .debug_abbrev     PROGBITS        0000000000000000 00156e 000187 00      0   0  1
  [ 5] .debug_aranges    PROGBITS        0000000000000000 001700 000060 00      0   0 16
  [ 6] .debug_line       PROGBITS        0000000000000000 001760 000292 00      0   0  1
  [ 7] .debug_str        PROGBITS        0000000000000000 0019f2 0001aa 01  MS  0   0  1
  [ 8] .debug_line_str   PROGBITS        0000000000000000 001b9c 000098 01  MS  0   0  1
  [ 9] .comment          PROGBITS        0000000000000000 001c34 000012 01  MS  0   0  1
  [10] .riscv.attributes RISCV_ATTRIBUTES 0000000000000000 001c46 000039 00      0   0  1
  [11] .debug_frame      PROGBITS        0000000000000000 001c80 0000d0 00      0   0  8
  [12] .symtab           SYMTAB          0000000000000000 001d50 000270 18     13  17  8
  [13] .strtab           STRTAB          0000000000000000 001fc0 000080 00      0   0  1
  [14] .shstrtab         STRTAB          0000000000000000 002040 00009e 00      0   0  1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  D (mbind), p (processor specific)

There are no section groups in this file.

Program Headers:
  Type           Offset   VirtAddr           PhysAddr           FileSiz  MemSiz   Flg Align
  RISCV_ATTRIBUT 0x000000 0x0000000000000000 0x0000000000000000 0x000000 0x000000 R   0x8
  LOAD           0x001000 0x0000000080080000 0x0000000000000000 0x000260 0x000260 RWE 0x1000

 Section to Segment mapping:
  Segment Sections...
   00
   01     .text .bss
...

Also, when I use objcopy to create a binary from the new elf files, gnuexample.elf produces correct and identical files when running both

riscv64-unknown-elf-objcopy -Obinary gnuexample.elf gnugnuexample.bin

and

llvm-objcopy -Obinary gnuexample.elf llvmgnuexample.bin

whereas creating a binary from llvmexample.elf produces wildly different results, with both GNU and LLVM seemingly interpreting some symbols' virtual address as its physical address (or the other way around, not sure) and producing binaries that are 0x80080270 bytes large.

EDIT: Edited incorrect information, was reading the wrong file, sorry.

[jh7370]

Thanks for the info - it should help someone take a look at this. Unfortuantely, that someone isn't going to be me due to other commitments.

[Kimplul]

I think I found a fix to my initial issue:

diff --git a/llvm/lib/ObjCopy/ELF/ELFObject.cpp b/llvm/lib/ObjCopy/ELF/ELFObject.cpp
index b0bf1402e..84e15a6c2 100644
--- a/llvm/lib/ObjCopy/ELF/ELFObject.cpp
+++ b/llvm/lib/ObjCopy/ELF/ELFObject.cpp
@@ -2645,8 +2645,7 @@ Error BinaryWriter::finalize() {
   uint64_t MinAddr = UINT64_MAX;
   for (SectionBase &Sec : Obj.allocSections()) {
     if (Sec.ParentSegment != nullptr)
-      Sec.Addr =
-          Sec.Offset - Sec.ParentSegment->Offset + Sec.ParentSegment->PAddr;
+      Sec.Addr = Sec.Addr - Sec.ParentSegment->VAddr + Sec.ParentSegment->PAddr;
     if (Sec.Type != SHT_NOBITS && Sec.Size > 0)
       MinAddr = std::min(MinAddr, Sec.Addr);
   }

From what I can understand, the existing code uses Sec.Addr as a temporary LMA, but the calculation uses section/segment offsets instead of addresses. The result is generally identical, but in my situation the offset and address have different alignments and the result differs. For reference, GNU binutils uses addresses in their calculations, see newsect->lma in

https://sourceware.org/git/?p=binutils-gdb.git;a=blob;f=bfd/elf.c;h=f046994e3a87fab4aa8fd68ca4fdc623f6118248;hb=HEAD#l1181

However, this only applies when writing directly to a binary, and trying to write an intermediate ELF file still shows the same issues. I think that's a separate bug in the ELF writer.

[jh7370]

I'm not really well-versed in what should be happening here, so it's probably better someone like @MaskRay takes a look at this?

[MaskRay]

Consider modifying your examples to use readelf -SW. The wider output is generally easier to read. Can you upload example.elf somewhere? It's difficult to analyze without the raw input.

[Kimplul]

I updated the examples, thanks for the tip, and I have a copy of the ELF file here:

https://github.com/Kimplul/misc/blob/main/example.elf

[MaskRay]

For a SHT_NOBITS section like .bss, its sh_offset is typically not aligned by sh_addralign. If it is converted to SHT_PROGBITS by --set-section-flags .bss=alloc,contents, the GNU behavior is to align it when computing the output size for -O binary.
I do not feel strongly about this behavior and whether we should match. If we should match, https://reviews.llvm.org/D128961 is a candidate patch.

@Kimplul Do you have a real world use case for this behavior?

[Kimplul]

At least what I used it for was creating a single binary blob that contains a program and immediately following the program a payload. This allows me to pretty easily reuse the program and attach different payloads to the image (with some header data for length etc.) without having to recompile the program for each payload and allows fairly easy handling since it's all in a single file, so easy to flash into some ROM.

Not necessarily saying it's the best way to handle things but has so far worked pretty good for my use case.