Skip to content

Implements Linux-style execve de_thread semantics and hands non-leader execs to the leader - #306

Merged
jserv merged 3 commits into
mainfrom
threaded-execve
Aug 18, 2026
Merged

Implements Linux-style execve de_thread semantics and hands non-leader execs to the leader#306
jserv merged 3 commits into
mainfrom
threaded-execve

Conversation

@jserv

@jserv jserv commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Summary by cubic

Implements Linux-style execve teardown with leader handoff so new images start single‑threaded and we never reset guest memory under live threads. Also fixes a clone region lookup race and adds per‑test hang stack sampling for timeouts.

  • Implements de_thread and hands non‑leader execve to the leader; replaces ad‑hoc checks with thread_stop_requested and adds thread_wake_all_blocked; guest teardown and exec use it to wake futex, poll, vCPU, and internal waiters.
  • Futex and timers: waits now watch thread_stop_requested and advance only ITIMER_REAL during host waits to avoid charging virtual timers; bounded backoff loops materialize SIGALRM promptly.
  • Poll/inotify/FUSE/time: finite waits run in slices and always carry the wakeup pipe; blocking reads and sleeps exit on thread_stop_requested instead of waiting out guest deadlines.
  • I/O and locks: introduces io_retry_backoff and bounded retries; converts blocking semop, flock, F_SETLKW, and O_WRONLY FIFO open into poll‑with‑retry so teardown can interrupt without parking in un‑wakeable host calls.
  • Fork interaction: serializes de_thread with the fork barrier/snapshot; thread_quiesce_siblings refuses new quiet windows once teardown is pending, callers abort with EINTR; teardown joins are bounded and avoid double‑teardown.
  • Concurrency fix: clone’s resolve_clone_stack_range now reads guest regions under mmap_lock to stop races with concurrent mmap/munmap.
  • Tests/harness: adds test-threaded-exec; integrates a watchdog stack sampler to capture hangs near timeouts; Makefile builds the new test.

Written for commit 5ca06a8. Summary will update on new commits.

Review in cubic

@jserv jserv changed the title Threaded execve Implements Linux-style execve de_thread semantics and hands non-leader execs to the leader Aug 17, 2026
cubic-dev-ai[bot]

This comment was marked as resolved.

cubic-dev-ai[bot]

This comment was marked as resolved.

cubic-dev-ai[bot]

This comment was marked as resolved.

jserv added 3 commits August 18, 2026 14:41
resolve_clone_stack_range reads the guest region array to find the
mapping a clone's child stack falls in. Any concurrent mmap or munmap
rewrites that array while holding mmap_lock, and clone never takes it,
so the read races g->regions and g->nregions.

ThreadSanitizer reports it as soon as one thread allocates while another
clones, which no test did until tests/test-threaded-exec.c gave its
workers an mmap loop. Neither caller holds a lock at that point and
mmap_lock is order 1, the outermost, so taking it there introduces no
inversion.
A hang that only reproduces under suite load is otherwise reported as a
bare "timeout after Ns" with nothing to diagnose. Both entry points arm
a watchdog around every invocation: it samples the live process shortly
before timeout(1) kills it, and the caller keeps the output only when
the watchdog actually fired.

The watchdog polls a sentinel rather than being killed by pid. Lanes
that spawn hundreds of short-lived processes can recycle a pid between
the watchdog exiting and a kill landing, which would turn this into a
random SIGTERM at an unrelated process.

The sentinel and the working file are named per arm. Sharing them would
force the passing path to block until the watchdog exited, because one
still symbolizing would otherwise write into the next test's file, and
every passing test would pay the remainder of a collection that is then
discarded. With distinct names a straggler can only touch its own, so
only the timed-out path waits, and only because the move needs a
finished file.

The watchdog re-checks its sentinel immediately before writing, and
removes its own output when the test it was watching has already been
reaped. Its collection ends by appending, which would otherwise recreate
a path the caller had just removed and leave it behind whenever no later
arm swept it.

run_timeout stays out of it: its callers pass their own cap and the
coreutils suite expects rc=124 from the guest's own timeout(1), so a 124
there does not mean the harness watchdog fired.
Linux de_thread() destroys every sibling before mapping the new image.
elfuse has to do the same before guest_reset zeroes the memory those
siblings are still running on, or they resume into a zeroed image.

thread_exec_de_thread runs in the post-failure region of sys_execve,
before the credential commit and the CLOEXEC sweep, so siblings wind
down against the old image. A sibling that outlives its bounded join
makes it return non-zero, and sys_execve takes its post-PNR fatal exit
rather than resetting guest memory under a live thread.

The leader is never a de_thread target, because its run loop returning
is what destroys the guest. A non-leader execve is handed to it: the
requester publishes the syscall arguments and blocks, the leader runs
the whole of sys_execve on its own vCPU, and the requester dies as a
sibling. The new image therefore always sees gettid() == getpid() and
Threads: 1.

For that join to terminate, every blocking wait a guest thread can
enter has to be reachable by a teardown wake, which is most of this
commit:

- poll and ppoll carry the wakeup pipe on every wait rather than only
  an indefinite one, and a finite wait runs to its deadline in slices
  that re-check the interrupt conditions.
- A blocking O_WRONLY FIFO open polls the non-blocking form instead of
  parking in the host until a reader arrives.
- F_SETLKW polls F_SETLK, decoding the guest's struct flock once before
  the loop so another thread cannot switch which region is locked
  underneath the wait.
- semop polls with IPC_NOWAIT forced onto a copy. When the set does not
  apply, repeating the walk the kernel makes before it blocks says which
  operation stopped it, so a set mixing IPC_NOWAIT with blocking
  operations keeps its per-operation semantics without ever entering a
  blocking host call. The walk answers SEMOP_BLOCKER_UNKNOWN when it
  cannot read the values, which a set granting alter but not read
  permission does, and the caller waits rather than inventing a refusal
  it cannot back up.
- The io and futex retry waits materialize an expired ITIMER_REAL
  before sleeping, so an alarm that fires during a contended lock is
  not delayed until the lock is acquired. Only ITIMER_REAL: Linux
  charges the other two to CPU time, and a thread parked in a host call
  spends none.

An execve teardown and a fork snapshot must not overlap. A sibling
parked in the fork barrier owes the forker its quiet until the copy
finishes, because unlike Linux, whose fork() takes a copy-on-write
snapshot later writes cannot reach, the copy here is not atomic against
a running thread. But a barrier that never releases strands that
sibling and the join reports it as one that refused to leave.

The two are serialized rather than raced. de_thread waits for an open
window to close before it publishes the teardown, and
thread_quiesce_siblings refuses to arm a new one once it has, with its
callers abandoning the operation the quiet was for. Both sides of that
handshake run under thread_lock, so the thread that releases the
barrier is never one the teardown is waiting on, and the barrier blocks
with no timer and no escape branch. The refusals return through the
existing cleanup labels: sys_clone has already spawned its child by
then, so a bare return would leak its socketpair and leave the child a
zombie nothing reaps.

One wait is still unreachable and recorded where it lives: the read
side of a blocking FIFO open, which macOS gives nothing to poll on in
any state.
@jserv
jserv merged commit 6ffc34a into main Aug 18, 2026
14 checks passed
@jserv
jserv deleted the threaded-execve branch August 18, 2026 06:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant