Anonymous mmap + mprotect(PROT_EXEC) bypasses ELF patch-cache exec interception

[dywongcloud]

Summary

Anonymous memory mappings can be transitioned from writable to executable without passing through the ELF patch-cache enforcement path.

The current executable-memory interception appears to depend on file_mappings, which is populated only by file-backed mappings created through do_mmap_file. Anonymous mappings created through do_mmap_anonymous → do_mmap_no_init are not represented in that cache. As a result, an anonymous RW mapping can later be changed to RX with sys_mprotect, bypassing maybe_patch_on_mprotect_exec.

Severity	Bug	Fix
CRITICAL	Full sandbox escape: mmap(RW,ANON) → write a raw syscall byte → mprotect(RX) ran unmediated on the host (only file-backed exec segments were ever scanned)	Reused the existing apply_trap_fallback byte-scanner for any exec-transition not covered by a tracked file mapping. Verified against Node's V8 JIT — correct, costs ~44ms on Node startup, an accepted tradeoff for closing a real escape
CRITICAL	Guest-controlled ASLR leak / control-flow-hijack: rt_sigreturn copied a guest-supplied rip/rsp straight into the resume context, unvalidated	Wired in sanitize_for_user_return() (already used by the lvbs/snp platforms) at the single guest-resume choke point
CRITICAL	Arbitrary host munmap: mremap's size arithmetic wrapped, letting a guest unmap any address it chose	checked_add, reject instead of wrap
HIGH	Ordinary dup2 to a modest fd number hit a hard assert! (undocumented internal bound, independent of the real RLIMIT_NOFILE check)	Graceful decline → ENOMEM/EMFILE
HIGH	mmap with a large length caused wraparound panics	checked_add throughout
HIGH	The sole network worker thread crashed (killing networking for every guest) on ordinary EAGAIN backpressure from the TUN device — no attack needed	Added SendError::WouldBlock, drop the packet like a real NIC driver would
LOW	Unguarded subtraction in AF_UNIX sockaddr encoding	Bounds check matching the sibling code path

Impact

This creates a sandbox escape path for executable anonymous memory.

A workload can allocate anonymous writable memory, populate it, and then call mprotect(..., PROT_EXEC) to make that region executable. Because the region was never registered as a file-backed mapping, the mprotect exec-interception logic does not patch or validate it.

This does not require JIT-specific behavior. It only requires ordinary memory mapping and protection syscalls.

Affected Flow

Current flow appears to be:

do_mmap_anonymous
  → do_mmap_no_init
    → anonymous mapping created
    → no ELF patch-cache interaction
    → no file_mappings entry

Then later:

sys_mprotect(PROT_EXEC)
  → maybe_patch_on_mprotect_exec
    → lookup in file_mappings
    → no entry found for anonymous mapping
    → mapping becomes executable without patch/cache enforcement

By contrast, file-backed executable mappings go through:

do_mmap_file
  → file_mappings populated
  → sys_mprotect(PROT_EXEC)
  → maybe_patch_on_mprotect_exec can inspect/patch

Expected Behavior

Any transition to executable memory should be mediated consistently, regardless of whether the mapping originated from:

• file-backed mmap
• anonymous mmap
• anonymous RW memory later converted with mprotect
• other mapping paths that can eventually become executable

Anonymous RW → RX transitions should either be denied, patched, validated, or routed through equivalent executable-memory policy enforcement.

Actual Behavior

Anonymous mappings are not tracked in file_mappings, and maybe_patch_on_mprotect_exec only checks that structure. Therefore anonymous memory can become executable without ELF patch-cache enforcement.

Suggested Fix

Treat executable permission transitions as the enforcement boundary, not only file-backed mapping creation.

Possible approaches:

1. Track anonymous mappings separately and require sys_mprotect(PROT_EXEC) to enforce policy for them.
2. Extend maybe_patch_on_mprotect_exec so it handles non-file-backed mappings explicitly.
3. Deny PROT_EXEC on anonymous mappings unless the sandbox has an explicit capability allowing executable anonymous memory.
4. Centralize all executable-memory transitions behind one policy function used by both mmap(PROT_EXEC) and mprotect(PROT_EXEC).
5. Add tests covering anonymous RW → RX transitions.

Regression Test Cases

Add tests for:

• anonymous mmap(PROT_READ | PROT_WRITE) followed by mprotect(PROT_READ | PROT_EXEC)
• anonymous mmap(PROT_READ | PROT_WRITE | PROT_EXEC) if supported
• file-backed mapping behavior still going through the patch path
• mixed mappings where only part of a region transitions to executable
• repeated RW → RX → RW → RX transitions

Security Severity

High / critical, depending on the sandbox threat model.

The issue bypasses executable-memory mediation using ordinary syscall behavior and does not require JIT support.

#1007

[CvvT]

Hi @dywongcloud Thanks for the report. We don't consider JIT as sandbox escape since there are legit use cases for that. The patching path is only meant to intercept syscalls and dispatch them through LiteBox (and we assume JIT code does not issue syscalls directly). We rely on seccomp to enforce security guarantee (if it is running on Linux userland).