`core::fmt::Write::write_fmt` causes UB when being relocated at runtime even when code-model is `pic`

[phip1611]

TL;DR: In this thread, I show how machine code emitted for core::fmt::write always jumps to the link address of a certain instruction although the code should be position independent (code-model is pie).

I'm working on a small x86 multiboot2 chainloader with position-independent code. The ELF consists of a small entry written in assembly (that sets up the stack) and rust code for more logic that is compiled/linked with relocation-model=pie. (I've also tried =pie but there was no difference.)

The ELF file a static ELF executable with one single LOAD segment. The Multiboot2 bootloader (GRUB) loads this file with a forced relocation (for example: pushed from 2 MiB to 6 MiB) into physical memory when the CPU is in Multiboot2's I386 machine state.

The assembly routine finds the relocation offset, sets the stack, and jumps to the high-level rust code. While some Rust functionality work properly (such as calling functions, doing calculating, printing text via an x86 I/O port) in the relocated binary, I noticed that

let x = format_args!("hello");
Printer.write_fmt(x); // small abstraction over QEMUs debug con device

causes undefined behavior and weird affects. A weird loop is started where the code above that line is executed again and again.

Interestingly, Printer.write_str("foo") works perfectly fine.

If I disable the relocation during runtime entirely, also core::fmt::Write::write_fmt works as expected and the string is printed to the screen. I had a look with an experienced colleague of mine and we are confident that my stack setup (even in the relocated case) is valid. Hence, the issue needs to be something else. From looking at the objdump, I'm certain that the code indeed uses relative addresses and calls.

I expected to see that the code is in fact relocatable in physical address space.

Instead, many functionality works but core::fmt::Write::write_fmt fails when the static ELF binary is relocated during runtime.

Meta

I'm using Rust in version nightly-2023-06-28. I'm using the following compiler spec JSON:

{
    "llvm-target": "i686-unknown-none",
    "data-layout": "e-m:e-i32:32-f80:128-n8:16:32-S128-p:32:32",
    "arch": "x86",
    "target-endian": "little",
    "target-pointer-width": "32",
    "target-c-int-width": "32",
    "os": "none",
    "executables": true,
    "linker-flavor": "ld.lld",
    "linker": "rust-lld",
    "panic-strategy": "abort",
    "disable-redzone": true,
    "features": "+soft-float,-x87,-mmx,-sse,-sse2,-sse3,-ssse3,-sse4.1,-sse4.2,-avx,-avx2,-fma,-3dnow,-3dnowa"
}

[Noratrieb]

Could you manually verified that the relocations inside the rust executable are messed up? Could you also include some sort of reproduction so the people can test it themselves?

[phip1611]

TL;DR: I'm sure there is a miscompilation/a bug in the compiler.

I do have a repository where people can test this out:

https://github.com/phip1611/relocatable-multiboot2-chain-loader/tree/poc-rust-issue-113207

If you have nix installed, running everything should be as easy as:

$ nix-shell --run "make && make integration-test". The Rust toolchain is pinned using a rustup-toolchain.toml file.

This starts my loader (=the rust binary I build) in a QEMU VM. GRUB2 chainloads the loader via multiboot2. The Rust binary is supposed to have only position-independent code and is linked at 2M. GRUB relocates the binary to 6M at runtime in physical memory.

To check if you have the right binary after running make, here's the sha256 hash:

sha256sum chainloader/target/x86-unknown-none/release/bin 
a2881d9545a1a969b278cc663fb7225b1f04a49539cf49195f809df4866b3ad1  chainloader/target/x86-unknown-none/release/bin

Hint: I'd like to mention that GRUB relocates my binary in "dumb"/simple way: There is no global offset table magic involved. The binary is just moved as simple as it can be in physical memory.

I investigated the problem further with QEMU and I know where the bug happens, but not why. I think, there is a mis-compilation?

Until the IP reaches address 0x2004cb (or 0x6004cb, which is the relocation in the setup of my example), everything is perfectly fine and works during runtime, as there is no format_args! code path involved. The relocation is no problem to the binary and the output is as expected. However, a few instructions later at link address 0x2015b0 (or 0x6015b0 at runtime), we find the following code:

  201599:       73 23                   jae    2015be <core::fmt::write+0x240>
  20159b:       8b 00                   mov    (%eax),%eax
  20159d:       8b 4c 24 2c             mov    0x2c(%esp),%ecx
  2015a1:       8b 5c 24 04             mov    0x4(%esp),%ebx
  2015a5:       ff 74 f0 04             push   0x4(%eax,%esi,8)
  2015a9:       ff 34 f0                push   (%eax,%esi,8)
  2015ac:       ff 74 24 30             push   0x30(%esp)
  2015b0:       ff 51 0c                call   *0xc(%ecx)  <-- this causes the bug
  2015b3:       83 c4 0c                add    $0xc,%esp

This belongs to the compiled code of core::fmt::write. During runtime/after relocation to 6M, this lives at 0x601599 (and so on). The code that runs during runtime at 0x6015b0 jumps back to 0x2002df, hence, the link address of a certain instruction, in any case. I think there is a mis-compilation here, as Rust did not give me position independent code, when I requested it to do so.

At link address 0x2002df, the code of <<&mut W as core::fmt::Write>::write_str is linked.

During runtime after relocation, there are a bunch of no-ops at 0x2002df. As a consequence, the CPU executes these noops and increases the IP, until the IP reaches the actual code of my loader at 6M eventually. The normal code path is executed again until the broken code is reached. And again and again.. This perfectly explains why I see the same output over and over again, if the bug occurs.

If I remove

    let x = format_args!("hello");
    let _ = Printer.write_fmt(x);

from chainloader/bin/src/main.rs, then the relocated binary works as expected.

I've appended a screencast of a GDB session right before the bug happens. The code here is executing at link address 0x2015b0, as shown above in the objdump output. (Hint: Somehow GDB shows x86 registers with r as prefix. However, we are in 32-bit mode.)

Screencast.from.2023-07-03.09-00-08.webm

[joboet]

Position independent code still requires you to apply ELF relocations in the loader. Normally, the compiler can just use RIP-relative addressing (this is why the rest of the binary works). But when a pointer to static memory is loaded from static memory (as is the case for format_args), it cannot do that, so it emits a relocation entry into the binary and leaves it to the loader to fix up the pointer.

[phip1611]

It seems like no relocation information are emitted for my binary - maybe I'm missing something as I never worked with ELF relocations so far. readelf -r tells there are no relocations.

I did the following modification to my linker script:

diff --git a/chainloader/bin/link.ld b/chainloader/bin/link.ld
index ecd8e03..e9d490c 100644
--- a/chainloader/bin/link.ld
+++ b/chainloader/bin/link.ld
@@ -15,6 +15,7 @@ PHDRS
      */
 
     loader     PT_LOAD FLAGS(7); /* 0b111 - read + write + execute */
+    dynamic PT_DYNAMIC;
 }
 
 SECTIONS {
@@ -22,6 +23,8 @@ SECTIONS {
     /* Binary is relocatable. Note: We want a 2 MiB alignment for huge pages. */
     .loader 2M : AT(2M) ALIGN(4K)
     {
+        *(.got .got.*)
+
         KEEP(*(.multiboot2_header));
 
         *(.text .text.*)
@@ -35,12 +38,42 @@ SECTIONS {
         *(.data .data.*)
     } : loader
 
+    /* I used `ld --verbose` to get these output sections. */
+    .rela.dyn       :
+        {
+          *(.rela.init)
+          *(.rela.text .rela.text.* .rela.gnu.linkonce.t.*)
+          *(.rela.fini)
+          *(.rela.rodata .rela.rodata.* .rela.gnu.linkonce.r.*)
+          *(.rela.data .rela.data.* .rela.gnu.linkonce.d.*)
+          *(.rela.tdata .rela.tdata.* .rela.gnu.linkonce.td.*)
+          *(.rela.tbss .rela.tbss.* .rela.gnu.linkonce.tb.*)
+          *(.rela.ctors)
+          *(.rela.dtors)
+          *(.rela.got)
+          *(.rela.bss .rela.bss.* .rela.gnu.linkonce.b.*)
+          *(.rela.ldata .rela.ldata.* .rela.gnu.linkonce.l.*)
+          *(.rela.lbss .rela.lbss.* .rela.gnu.linkonce.lb.*)
+          *(.rela.lrodata .rela.lrodata.* .rela.gnu.linkonce.lr.*)
+          *(.rela.ifunc)
+        }  : loader
+      .rela.plt       :
+        {
+          *(.rela.plt)
+          PROVIDE_HIDDEN (__rela_iplt_start = .);
+          *(.rela.iplt)
+          PROVIDE_HIDDEN (__rela_iplt_end = .);
+        } : loader
+      .relr.dyn : { *(.relr.dyn) }  : loader
+    .plt            : { *(.plt) *(.iplt) } : loader
+    .plt.got        : { *(.plt.got) } : loader
+    .plt.sec        : { *(.plt.sec) } : loader
+    .dynamic : { *(.dynamic) } :data :dynamic
+
     /DISCARD/ :
     {
         *(.note.*)
         *(.comment .comment.*)
         *(.eh_frame*)
-        *(.got .got.*)
     }
 
 }

The difference of readelf is the following:

3,4c3,4
< Entry point 0x200098
< There is 1 program header, starting at offset 52
---
> Entry point 0x2000a8
> There are 2 program headers, starting at offset 52
8c8,10
<   LOAD           0x001000 0x00200000 0x00200000 0x05188 0x05188 RWE 0x1000
---
>   LOAD           0x001000 0x00200000 0x00200000 0x05198 0x05198 RWE 0x1000
>   DYNAMIC        0x000000 0x00000000 0x00000000 0x00000 0x00000 R   0
> readelf: Error: no .dynamic section in the dynamic segment
12a15
>    01     

*(.got .got.*) adds 16 bytes of 0xcc values to the binary.

By the way: in any way, readelf -r tells there are no relocations.

[phip1611]

By switching from ldd.ld to ld in the compiler spec (my compiler spec definition can be found in the most top post) and printing the linker memory map (-M=/tmp/app.mem), I found that there are a lot of .data.rel.ro sections that I silently linked together with the rest of .data into the loader segment. I don't know how helpful this information might be.. The dynamic program header keeps staying empty.

Here's an excerpt from the memory map during linkage of GNU ld when I use my own linker script:

Discarded input sections

 .rodata..Lanon.fb9cc2090afbfef5494a6ad91fb73b50.501
                0x00000000        0x3 /home/pschuster/dev/other/relocatable-multiboot2-chain-loader/chainloader/target/x86-unknown-none/release/deps/bin-1e93604e6a8333f1.bin.f860c92f8c55bdb3-cgu.0.rcgu.o
 .note.GNU-stack
                0x00000000        0x0 /home/pschuster/dev/other/relocatable-multiboot2-chain-loader/chainloader/target/x86-unknown-none/release/deps/bin-1e93604e6a8333f1.bin.f860c92f8c55bdb3-cgu.0.rcgu.o

Memory Configuration

Name             Origin             Length             Attributes
*default*        0x00000000         0xffffffff

Linker script and memory map

LOAD /tmp/rustcxsBFJA/symbols.o
LOAD /home/pschuster/dev/other/relocatable-multiboot2-chain-loader/chainloader/target/x86-unknown-none/release/deps/bin-1e93604e6a8333f1.bin.f860c92f8c55bdb3-cgu.0.rcgu.o
LOAD /home/pschuster/dev/other/relocatable-multiboot2-chain-loader/chainloader/target/x86-unknown-none/release/deps/libcompiler_builtins-1e363e06abeb7127.rlib

.multiboot2_header
                0x00200000       0x98
 *(.multiboot2_header)
 .multiboot2_header
                0x00200000       0x98 /home/pschuster/dev/other/relocatable-multiboot2-chain-loader/chainloader/target/x86-unknown-none/release/deps/bin-1e93604e6a8333f1.bin.f860c92f8c55bdb3-cgu.0.rcgu.o

.rel.dyn        0x00200098        0x0
 .rel.got       0x00200098        0x0 /tmp/rustcxsBFJA/symbols.o
 .rel.iplt      0x00200098        0x0 /tmp/rustcxsBFJA/symbols.o

.data.rel.ro    0x00200098      0x248
 *(.data.rel.ro.local* .gnu.linkonce.d.rel.ro.local.*)
 *(.data.rel.ro .data.rel.ro.* .gnu.linkonce.d.rel.ro.*)
 .data.rel.ro..Lanon.45b9a0693cc52d86b71b55c0a4f0cf49.0
                0x00200098       0x18 /home/pschuster/dev/other/relocatable-multiboot2-chain-loader/chainloader/target/x86-unknown-none/release/deps/bin-1e93604e6a8333f1.bin.f860c92f8c55bdb3-cgu.0.rcgu.o
 
  ...
                0x002002c0       0x10 /home/pschuster/dev/other/relocatable-multiboot2-chain-loader/chainloader/target/x86-unknown-none/release/deps/bin-1e93604e6a8333f1.bin.f860c92f8c55bdb3-cgu.0.rcgu.o
 .data.rel.ro..Lanon.fb9cc2090afbfef5494a6ad91fb73b50.503
                0x002002d0       0x10 /home/pschuster/dev/other/relocatable-multiboot2-chain-loader/chainloader/target/x86-unknown-none/release/deps/bin-1e93604e6a8333f1.bin.f860c92f8c55bdb3-cgu.0.rcgu.o

.rela.dyn
 *(.rela.init)
 *(.rela.text .rela.text.* .rela.gnu.linkonce.t.*)
 *(.rela.fini)
 *(.rela.rodata .rela.rodata.* .rela.gnu.linkonce.r.*)
 *(.rela.data .rela.data.* .rela.gnu.linkonce.d.*)
 *(.rela.tdata .rela.tdata.* .rela.gnu.linkonce.td.*)
 *(.rela.tbss .rela.tbss.* .rela.gnu.linkonce.tb.*)
 *(.rela.ctors)
 *(.rela.dtors)
 *(.rela.got)
 *(.rela.bss .rela.bss.* .rela.gnu.linkonce.b.*)
 *(.rela.ldata .rela.ldata.* .rela.gnu.linkonce.l.*)
 *(.rela.lbss .rela.lbss.* .rela.gnu.linkonce.lb.*)
 *(.rela.lrodata .rela.lrodata.* .rela.gnu.linkonce.lr.*)
 *(.rela.ifunc)

.rela.plt       0x002002e0        0x0
 *(.rela.plt)
                [!provide]                        PROVIDE (__rela_iplt_start = .)
 *(.rela.iplt)
                [!provide]                        PROVIDE (__rela_iplt_end = .)

.relr.dyn
 *(.relr.dyn)

.plt            0x002002e0        0x0
 *(.plt)
 *(.iplt)
 .iplt          0x002002e0        0x0 /tmp/rustcxsBFJA/symbols.o

.plt.got
 *(.plt.got)

.plt.sec
 *(.plt.sec)

.dynamic
 *(.dynamic)

.got            0x002002e0        0xc
 *(.got.plt)
 .got.plt       0x002002e0        0xc /tmp/rustcxsBFJA/symbols.o
                0x002002e0                _GLOBAL_OFFSET_TABLE_
 *(.igot.plt)
 .igot.plt      0x002002ec        0x0 /tmp/rustcxsBFJA/symbols.o
 *(.got)
 .got           0x002002ec        0x0 /tmp/rustcxsBFJA/symbols.o
 *(.igot)

.loader         0x002002f0     0x4ea0
 *(.text .text.*)
 .text          0x002002f0       0x30 /home/pschuster/dev/other/relocatable-multiboot2-chain-loader/chainloader/target/x86-unknown-none/release/deps/bin-1e93604e6a8333f1.bin.f860c92f8c55bdb3-cgu.0.rcgu.o
                0x002002f0                start
  
  ....

 .text._ZN64_$LT$core..str..error..Utf8Error$u20$as$u20$core..fmt..Debug$GT$3fmt17hbba04a114ea0b8e7E
                0x00203018       0x93 /home/pschuster/dev/other/relocatable-multiboot2-chain-loader/chainloader/target/x86-unknown-none/release/deps/bin-1e93604e6a8333f1.bin.f860c92f8c55bdb3-cgu.0.rcgu.o
 *(.rodata .rodata.*)
 *fill*         0x002030ab        0x1 
 .rodata._ZN4core3fmt9Formatter7padding17hef30d55e26cdeb48E
                0x002030ac       0x10 /home/pschuster/dev/other/relocatable-multiboot2-chain-loader/chainloader/target/x86-unknown-none/release/deps/bin-1e93604e6a8333f1.bin.f860c92f8c55bdb3-cgu.0.rcgu.o
   
  ...

 .rodata._ZN4core7unicode12unicode_data15grapheme_extend7OFFSETS17hbf72781b6b8236b1E
                0x00203eb8      0x2d7 /home/pschuster/dev/other/relocatable-multiboot2-chain-loader/chainloader/target/x86-unknown-none/release/deps/bin-1e93604e6a8333f1.bin.f860c92f8c55bdb3-cgu.0.rcgu.o
 *(COMMON)
 *(.bss .bss.*)
 *(.data .data.*)
 *fill*         0x0020418f        0x1 
 .data          0x00204190     0x1000 /home/pschuster/dev/other/relocatable-multiboot2-chain-loader/chainloader/target/x86-unknown-none/release/deps/bin-1e93604e6a8333f1.bin.f860c92f8c55bdb3-cgu.0.rcgu.o

/DISCARD/
 *(.note.*)
 *(.comment .comment.*)
 *(.eh_frame*)
OUTPUT(/home/pschuster/dev/other/relocatable-multiboot2-chain-loader/chainloader/target/x86-unknown-none/release/deps/bin-1e93604e6a8333f1 elf32-i386)

When I use lld's default linker script, the readelf -r output also states that there are no relocations in this file. So I think, my linker script is fine.

[phip1611]

By adding the following patch to my project, I could indeed create an executable with relocation information.

(However, my planned project doesn't work then, as GRUB2 doesn't support to (chain)load binaries with relocs.

So, this is less of a bug and more a works as intended? It's a bit frustrating. I was very motivated already as some of the compiled code was indeed fully position independent without further relocation information.)

I'd prefer a compiler mode where rustc either guarantees:

• a full position independent exectuable withour reloc information, or
• a build failure, as the code one tried to compile can only work with reloc information (or static addresses)

diff --git a/chainloader/bin/link.ld b/chainloader/bin/link.ld
index ad93a8c..5394b8f 100644
--- a/chainloader/bin/link.ld
+++ b/chainloader/bin/link.ld
@@ -22,6 +22,7 @@ PHDRS
      */
 
     loader     PT_LOAD FLAGS(7); /* 0b111 - read + write + execute */
+    dynamic    PT_DYNAMIC FLAGS(6);
 }
 
 SECTIONS {
@@ -42,12 +43,21 @@ SECTIONS {
         *(.data .data.*)
     } : loader
 
+    .got : { *(.got .got.*) } : loader
+
+    .dynamic : { *(.dynamic .dynamic.*) } : loader
+
+    .interp : { *(.interp .interp.*) } : loader
+    .dynsym : { *(.dynsym .dynsym.*) } : loader
+    .dynstr : { *(.dynstr .dynstr.*) } : loader
+    .hash : { *(.hash .hash.*) } : loader
+    .interp : { *(.gnu.hash .gnu.hash.*) } : loader
+
     /DISCARD/ :
     {
         *(.note.*)
         *(.comment .comment.*)
         *(.eh_frame*)
-        *(.got .got.*)
     }
 
 }
diff --git a/chainloader/x86-unknown-none.json b/chainloader/x86-unknown-none.json
index d3e81d6..ef5a044 100644
--- a/chainloader/x86-unknown-none.json
+++ b/chainloader/x86-unknown-none.json
@@ -7,8 +7,10 @@
     "target-c-int-width": "32",
     "os": "none",
     "executables": true,
-    "linker-flavor": "ld.lld",
-    "linker": "rust-lld",
+    "linker-flavor": "ld",
+    "linker": "ld",
+    "position-independent-executables": true,
+    "static-position-independent-executables": true,
     "panic-strategy": "abort",
     "disable-redzone": true,
     "features": "+soft-float,-x87,-mmx,-sse,-sse2,-sse3,-ssse3,-sse4.1,-sse4.2,-avx,-avx2,-fma,-3dnow,-3dnowa"

[bjorn3]

I'd prefer a compiler mode where rustc either guarantees:

• a full position independent exectuable withour reloc information, or
• a build failure, as the code one tried to compile can only work with reloc information (or static addresses)

This would (almost) always produce a build error as every program links in libcore, which has several statics that contain references. And even in case of a PIC/PIE executable data sections that contain pointers need relocations to be applied at runtime. PIC/PIE only applies to code, not to data. For code relative addressing is possible if the target is in the same DSO, but data requires absolute pointers.

[DemiMarie]

I'd prefer a compiler mode where rustc either guarantees:

• a full position independent exectuable withour reloc information, or
• a build failure, as the code one tried to compile can only work with reloc information (or static addresses)

This would (almost) always produce a build error as every program links in libcore, which has several statics that contain references. And even in case of a PIC/PIE executable data sections that contain pointers need relocations to be applied at runtime.

Text relocations require that the code segment be temporarily made writable to perform relocation processing. This violates W^X, so modern toolchains do not emit them and Android forbids them outright.

PIC/PIE only applies to code, not to data. For code relative addressing is possible if the target is in the same DSO, but data requires absolute pointers.

Data needs relocations applied, but code does not. That said, I would not be surprised if missing data relocations could cause the problem @phip1611 observed. In short, GRUB does not support loading binaries that have been compiled as PIC/PIE, and instead requires that all code be linked at the base address it will be run from. This allows all relocation processing to be done by the linker before execution. Systems software that needs to be relocated will need to relocate itself after GRUB has loaded it.

[phip1611]

This would (almost) always produce a build error as every program links in libcore, which has several statics that contain references. And even in case of a PIC/PIE executable data sections that contain pointers need relocations to be applied at runtime. PIC/PIE only applies to code, not to data. For code relative addressing is possible if the target is in the same DSO, but data requires absolute pointers.

Data needs relocations applied

I see. My expectation was that having a single RWX LOAD segment, it is possible to have 100% position independent code and data accesses without additional relocation information in the ELF.

So to sum this up, I ended up in an unfortunate and uncommon toolchain situation where things somehow only worked as expected to a certain degree.

Would it be a sensible solution to add a compiler warning output such as the following?

if relocation_model == "pie" && !target.static_position_independent_executables {
    eprintln!("Binary will not have relocation information. A relocation during runtime might cause undefined behaviour.");
}