Why does this x86_64 self-modifying loop work on bare metal / QEMU, but Segfaults only on native Linux under high CPU load?

[studioframes]

🏷️ Discussion Type

Question

💬 Feature/Topic Area

Assembly / x86_64 Architecture

Body

Hey everyone,

I'm writing a lightweight x86_64 JIT engine, and I ran into a bizarre hardware/kernel-level issue that I can't wrap my head around.

I have a small routine that dynamically patches its own immediate values inside a loop:

section .text
global _start

_start:
    mov rbx, 10             ; Loop counter

.loop:
    ; Self-modifying instruction target
    mov rax, 0x1122334455667788  
patch_target equ $ - 8

    ; Modify the immediate value dynamically for next iteration
    add qword [rel patch_target], 1

    dec rbx
    jnz .loop

    ; Exit syscall
    mov rax, 60
    xor rdi, rdi
    syscall

The Problem:

1. I marked the .text segment as readable, writable, and executable (-z execstack / mprotect(PROT_READ | PROT_WRITE | PROT_EXEC)).
2. When running under QEMU or single-core VMs, it executes to completion perfectly.
3. On a native multi-core x86_64 Linux machine (Ubuntu 24.04), it randomly throws Segmentation fault (core dumped) or executes with stale data inside RAX on every ~5th run.

Is this an L1 Instruction Cache (i-cache) pipeline coherency issue where the CPU prefetches the unpatched instruction before add qword commits to memory, or am I violating alignment requirements for RIP-relative addressing on 64-bit ModR/M bytes?

Do I need an explicit MFENCE / SERIALIZE instruction here, or is Linux handling page fault permissions asynchronously?

Guidelines

• I have read and understood this category's guidelines before making this post.

[AntonAzer]

Hi @studioframes!

This issue comes down to two hardware-level phenomena on modern x86_64 CPUs: Store Buffer vs. IFU race conditions and Cache-Line Tearing.

1. Store Buffer vs. IFU (Stale Data)

The Instruction Fetch Unit (IFU) fetches from L1i and does not snoop the CPU's local Store Buffer. Under heavy CPU load, the updated immediate value written by add qword sits in the Store Buffer long enough for the speculatively prefetched mov rax to retire with stale data before the store drains into L1d to trigger an SMC Machine Clear.

2. Cache-Line Tearing (Segfaults)

Your 8-byte immediate (patch_target) is likely unaligned and straddles a 64-byte cache line boundary. Writes across cache lines are non-atomic. If the IFU fetches the instruction mid-store, it reads a corrupted ("Frankenstein") value into RAX. Dereferencing or jumping to this corrupted value triggers the Segmentation Fault.

Why QEMU works: QEMU (TCG) translates blocks sequentially and invalidates Translation Blocks synchronously, bypassing store buffer mechanics entirely.

---

How to Fix It

Option A: Avoid SMC in Tight Loops (Recommended)
Self-modifying code triggers severe hardware pipeline flushes. Instead of modifying instructions on every iteration, mutate the value in a register:

    mov rax, 0x1122334455667788
.loop:
    ; ... do work ...
    add rax, 1
    dec rbx
    jnz .loop

Option B: Alignment + Serialization (If SMC is required)

1. Align instructions with NOPs so patch_target never crosses a 64-byte boundary.
2. Drain the Store Buffer and serialize the pipeline using mfence + serialize (or cpuid on older CPUs):

.loop:
    mov rax, 0x1122334455667788
patch_target equ $ - 8

    add qword [rel patch_target], 1
    mfence
    serialize    ; or cpuid on older CPUs
    
    dec rbx
    jnz .loop