Dubious section file offset increases?

[shenhanc78]

Tried a simple binary, noticed in the transformed binary, section [24]->[25], file offset increases by 4M, thus greatly increases the transformed binary size.

Section Headers:
  [Nr] Name              Type            Address          Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            0000000000000000 000000 000000 00      0   0  0
  [ 1] .interp           PROGBITS        0000000000400238 000238 00001c 00   A  0   0  1
  [ 2] .note.ABI-tag     NOTE            0000000000400254 000254 000020 00   A  0   0  4
  [ 3] .hash             HASH            0000000000400278 000278 000024 04   A  5   0  8
  [ 4] .gnu.hash         GNU_HASH        00000000004002a0 0002a0 00001c 00   A  5   0  8
  [ 5] .dynsym           DYNSYM          00000000004002c0 0002c0 000060 18   A  6   1  8
  [ 6] .dynstr           STRTAB          0000000000400320 000320 00003f 00   A  0   0  1
  [ 7] .gnu.version      VERSYM          0000000000400360 000360 000008 02   A  5   0  2
  [ 8] .gnu.version_r    VERNEED         0000000000400368 000368 000020 00   A  6   1  8
  [ 9] .rela.dyn         RELA            0000000000400388 000388 000030 18   A  5   0  8
  [10] .rela.plt         RELA            00000000004003b8 0003b8 000018 18  AI  5  22  8
  [11] .init             PROGBITS        00000000004003d0 0003d0 000017 00  AX  0   0  4
  [12] .plt              PROGBITS        00000000004003f0 0003f0 000020 10  AX  0   0 16
  [13] .bolt.org.text    PROGBITS        0000000000400410 000410 000202 00  AX  0   0 16
  [14] .fini             PROGBITS        0000000000400614 000614 000009 00  AX  0   0  4
  [15] .rodata           PROGBITS        0000000000400620 000620 000008 00   A  0   0  4
  [16] .bolt.org.eh_frame_hdr PROGBITS        0000000000400628 000628 00003c 00   A  0   0  4
  [17] .bolt.org.eh_frame PROGBITS        0000000000400668 000668 000110 00   A  0   0  8
  [18] .init_array       INIT_ARRAY      0000000000600e00 000e00 000008 08  WA  0   0  8
  [19] .fini_array       FINI_ARRAY      0000000000600e08 000e08 000008 08  WA  0   0  8
  [20] .dynamic          DYNAMIC         0000000000600e10 000e10 0001e0 10  WA  6   0  8
  [21] .got              PROGBITS        0000000000600ff0 000ff0 000010 08  WA  0   0  8
  [22] .got.plt          PROGBITS        0000000000601000 001000 000020 08  WA  0   0  8
  [23] .data             PROGBITS        0000000000601020 001020 000010 00  WA  0   0  8
  [24] .tm_clone_table   PROGBITS        0000000000601030 001030 000000 00  WA  0   0  8
  [25] .bss              NOBITS          0000000000601030 4002c0 000008 00  WA  0   0  1
  [26] .text             PROGBITS        0000000000a00000 600000 00020b 00  AX  0   0 64
  [27] .eh_frame         PROGBITS        0000000000bfff50 7fff50 0001e0 00   A  0   0  8
  [28] .eh_frame_hdr     PROGBITS        0000000000c00130 800130 00003c 00   A  0   0  1
  [29] .comment          PROGBITS        0000000000000000 80016c 0000c1 01  MS  0   0  1
  [30] .symtab           SYMTAB          0000000000000000 800230 000600 18     31  45  8
  [31] .strtab           STRTAB          0000000000000000 800830 0001f9 00      0   0  1
  [32] .shstrtab         STRTAB          0000000000000000 800a29 000199 00      0   0  1
  [33] .note.bolt_info   NOTE            0000000000000000 800bc2 00009c 00      0   0  1

[maksfb]

We do this to align code at 2MB boundary to assist with iTLB misses (we do use 2MB pages for code). In this case it obviously doesn't help, but wouldn't hurt the performance either. Is there a reason the size increase is a concern? We can add a special mode to disable the alignment.

[shenhanc78]

Thanks. Yes, 2MB alignment is reasonable.What about just bumping the section "Address", leaving a hole between section addresses, but keep "Off" (the file offset) continuous, otherwise we have holes in the file and unnecessary increase file size.

[Nr] Name              Type            Address           Off     Size   ES Flg Lk Inf Al
  [ x]                 NULL            0000000000000000  000000  00010
  [ x+1]               NULL            0000000000200000  000010  01000

[tmsri]

I think this is just a bug, correct me if I am wrong:

[21] .data
PROGBITS 0000000000601018 001018 000010 00 0 0 8
[0000000000000003]: WRITE, ALLOC
[22] .tm_clone_table
PROGBITS 0000000000601028 001028 000000 00 0 0 8
[0000000000000003]: WRITE, ALLOC
[23] .bss
NOBITS 0000000000601028 4002c0 000008 00 0 0 1
[0000000000000003]: WRITE, ALLOC
[24] .text
PROGBITS 0000000000a00000 600000 00018b 00 0 0 64
[0000000000000006]: ALLOC, EXEC

The file offset of bss is 0x4002c0 but I see this with readelf -l

Program Headers:
Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
PHDR 0x400000 0x0000000000800000 0x0000000000800000 0x000230 0x000230 R 0x8
INTERP 0x000238 0x0000000000400238 0x0000000000400238 0x00001c 0x00001c R 0x1
[Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]
LOAD 0x000000 0x0000000000400000 0x0000000000400000 0x000600 0x000600 R E 0x200000
LOAD 0x000e40 0x0000000000600e40 0x0000000000600e40 0x0001e8 0x0001f0 RW 0x200000
LOAD 0x400000 0x0000000000800000 0x0000000000800000 0x400064 0x400064 R E 0x200000

Section to Segment mapping:
Segment Sections...
00
01 .interp
02 .interp .note.ABI-tag .hash .gnu.hash .dynsym .dynstr .gnu.version .gnu.version_r .rela.dyn .init .bolt.org.text .fini .rodata .bolt.org.eh_frame_hdr .bolt.org.eh_frame
03 .init_array .fini_array .dynamic .got .got.plt .data .bss
04 .text .eh_frame .eh_frame_hdr

bss is in the second segment whose file offset ends much before 0x4002c0, so .bss cannot be at this file offset. The virtual address for .bss however seems right which is more interesting and important than file offset. @shenhanc78 @maksfb

[maksfb]

The reason we have to keep 2MB offset in the file, as well as in memory, is explained at https://github.com/facebookincubator/BOLT/blob/a25bab5f0163f1cd3bf68481d3663141a1594ea4/src/RewriteInstance.cpp#L825. In short, it guarantees to work for loaders that expect the program header table to be in the first loadable segment. Overall, we get ~4MB increase because we place program header table first thing in the new segment, and then we have to align at 2MB again for .text.

If you use -no-huge-pages option, then BOLT will use 4K alignment instead of 2MB, and the file size increase will be smaller.

The .bss section offset should be fixed in the most recent version.