Stop faking SIGCHLD on a CLONE_THREAD exit - #361
Conversation
When the last CLONE_THREAD worker exited, forkipc.c raised the futex interrupt, which makes the next blocking call in the surviving thread return EINTR. The comment gave the reason: "In real Linux, child exit delivers SIGCHLD which interrupts futex_wait with -EINTR." Linux does not. clone(2) is explicit that a thread created with CLONE_THREAD sends no signal to its parent when it terminates; the thread group's exit signal goes to the process's parent, not to a sibling. Nothing is delivered, so a sibling's wait is not interrupted and its timeout is what ends it. Measured on Linux 6.18.44 through the qemu lane, a worker exits and the main thread then parks for 300 ms with nothing to wake it: call Linux elfuse before futex(FUTEX_WAIT) ETIMEDOUT 303ms EINTR 101ms ppoll 0 303ms EINTR 0ms epoll_pwait 0 311ms EINTR 302ms nanosleep 0 309ms 0 328ms No signal is pending on either side. nanosleep agrees because it does not read the flag, which is what identifies the flag as the mechanism rather than anything process-wide. Removing the request drops the fabricated errno and nothing else. The futex waiter is not woken by the interrupt: it wakes on its own 100 ms quantum and re-checks thread_stop_requested, its woken flag, expired itimers and queued signals, of which futex_interrupt_consume is one reason among several. wakeup_pipe_signal and thread_interrupt_all stay, so threads parked on the shared pipe and inside hv_vcpu_run still get the nudge this site exists to send. Teardown keeps its interrupt. All four callers that mean it go through thread_wake_all_blocked, where every thread really is leaving and the EINTR is one each of them has to see. This site was the only one that raised it while the process carried on running. The one-shot consume in 520568c is untouched. That fixed the flag staying set, which is why foot spun on EINTR forever; it did not ask whether the flag should have been set at all. tests/test-futex-no-phantom-eintr.c parks in each of the three calls above after a worker exit and fails on an early return. It fails three of seven before this change, passes after, and passes unchanged on the reference kernel. It lives in test-matrix.sh's run_unit_tests and not in tests/manifest.txt, per that file's scope note: every assertion is a timeout the guest can observe, so it is cross-checkable. The CLEARTID wait inside it is spelled out rather than reusing raw_futex_wait, which carries FUTEX_PRIVATE_FLAG. The exit-time wake is a plain futex wake, so a private wait never matches its key and the test hung on the reference kernel until it was a plain wait with a timeout.
There was a problem hiding this comment.
3 issues found across 4 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="tests/test-futex-no-phantom-eintr.c">
<violation number="1" location="tests/test-futex-no-phantom-eintr.c:89">
P2: When the worker exits while `burn_worker()` is blocked, this reap futex consumes the one-shot phantom EINTR before the calls under test run. Reap without a futex wait, or otherwise preserve the interrupt until the target wait.</violation>
</file>
<file name="src/runtime/futex.c">
<violation number="1" location="src/runtime/futex.c:587">
P3: The comment incorrectly says the one-shot flag gives every teardown thread an EINTR. Document that the atomic interrupt is consumed by one waiter, while teardown state wakes the others.</violation>
<violation number="2" location="src/runtime/futex.c:593">
P3: The file-header comment on futex_interrupt_requested (lines 66-67) is now stale: it still says the flag is 'Used to simulate SIGCHLD delivery when all CLONE_THREAD workers exit', which is exactly the behavior this PR removes and the new futex_interrupt_consume comment explicitly says is gone. Update that header comment to reflect that the interrupt is now raised only by teardown via thread_wake_all_blocked, so the two comments do not contradict each other.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| struct k_timespec quantum = {0, 100L * 1000 * 1000}; | ||
| long deadline = now_ms() + 5000; | ||
| while (__atomic_load_n((volatile int *) &ctid, __ATOMIC_ACQUIRE) != 0) { | ||
| raw_syscall6(__NR_futex, (long) &ctid, FUTEX_WAIT, r, (long) &quantum, |
There was a problem hiding this comment.
P2: When the worker exits while burn_worker() is blocked, this reap futex consumes the one-shot phantom EINTR before the calls under test run. Reap without a futex wait, or otherwise preserve the interrupt until the target wait.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At tests/test-futex-no-phantom-eintr.c, line 89:
<comment>When the worker exits while `burn_worker()` is blocked, this reap futex consumes the one-shot phantom EINTR before the calls under test run. Reap without a futex wait, or otherwise preserve the interrupt until the target wait.</comment>
<file context>
@@ -0,0 +1,186 @@
+ struct k_timespec quantum = {0, 100L * 1000 * 1000};
+ long deadline = now_ms() + 5000;
+ while (__atomic_load_n((volatile int *) &ctid, __ATOMIC_ACQUIRE) != 0) {
+ raw_syscall6(__NR_futex, (long) &ctid, FUTEX_WAIT, r, (long) &quantum,
+ 0, 0);
+ if (now_ms() > deadline)
</file context>
| * paths through thread_wake_all_blocked: every thread really is leaving, so the | ||
| * EINTR is one each of them has to see. Without the clear, the flag stays set |
There was a problem hiding this comment.
P3: The comment incorrectly says the one-shot flag gives every teardown thread an EINTR. Document that the atomic interrupt is consumed by one waiter, while teardown state wakes the others.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At src/runtime/futex.c, line 587:
<comment>The comment incorrectly says the one-shot flag gives every teardown thread an EINTR. Document that the atomic interrupt is consumed by one waiter, while teardown state wakes the others.</comment>
<file context>
@@ -583,13 +583,16 @@ void futex_interrupt_clear(void)
- * does, and the spinning main thread eventually faults in a code path the guest
- * never expects to reach.
+ * clears it, 0 otherwise. The interrupt is a one-shot edge, set by the teardown
+ * paths through thread_wake_all_blocked: every thread really is leaving, so the
+ * EINTR is one each of them has to see. Without the clear, the flag stays set
+ * and every subsequent epoll_pwait, ppoll, futex wait, etc. spins on EINTR
</file context>
| * paths through thread_wake_all_blocked: every thread really is leaving, so the | |
| * EINTR is one each of them has to see. Without the clear, the flag stays set | |
| * paths through thread_wake_all_blocked. Teardown state marks every thread as | |
| * leaving; the atomic interrupt itself is consumed by only one waiter. |
| * until execve clears it -- in foot's case it never does, and the spinning main | ||
| * thread eventually faults in a code path the guest never expects to reach. | ||
| * | ||
| * forkipc.c set it too, on the last clone-thread exit, for a SIGCHLD that |
There was a problem hiding this comment.
P3: The file-header comment on futex_interrupt_requested (lines 66-67) is now stale: it still says the flag is 'Used to simulate SIGCHLD delivery when all CLONE_THREAD workers exit', which is exactly the behavior this PR removes and the new futex_interrupt_consume comment explicitly says is gone. Update that header comment to reflect that the interrupt is now raised only by teardown via thread_wake_all_blocked, so the two comments do not contradict each other.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At src/runtime/futex.c, line 593:
<comment>The file-header comment on futex_interrupt_requested (lines 66-67) is now stale: it still says the flag is 'Used to simulate SIGCHLD delivery when all CLONE_THREAD workers exit', which is exactly the behavior this PR removes and the new futex_interrupt_consume comment explicitly says is gone. Update that header comment to reflect that the interrupt is now raised only by teardown via thread_wake_all_blocked, so the two comments do not contradict each other.</comment>
<file context>
@@ -583,13 +583,16 @@ void futex_interrupt_clear(void)
+ * until execve clears it -- in foot's case it never does, and the spinning main
+ * thread eventually faults in a code path the guest never expects to reach.
+ *
+ * forkipc.c set it too, on the last clone-thread exit, for a SIGCHLD that
+ * clone(2) does not send. That is gone; only a process actually tearing down
+ * fabricates an EINTR now.
</file context>
jserv
left a comment
There was a problem hiding this comment.
Refine tests/test-futex-no-phantom-eintr.c to be more consistent with existing naming schemes.
When the last
CLONE_THREADworker exits,forkipc.craises the futexinterrupt, which makes the next blocking call in the surviving thread return
EINTR. The comment gave the reason: "In real Linux, child exit deliversSIGCHLD which interrupts futex_wait with -EINTR."
Linux does not.
clone(2)is explicit that a thread created withCLONE_THREADsends no signal to its parent when it terminates; the threadgroup's exit signal goes to the process's parent, not to a sibling. Nothing
is delivered, so the sibling's wait runs to its timeout.
Reproduction
Clone a
CLONE_THREADworker that exits at once, wait for theCLONE_CHILD_CLEARTIDstore, then park for 300 ms with nothing to wake thecaller.
tests/test-futex-no-phantom-eintr.cis that program; the threerows below are what it observes.
futex(FUTEX_WAIT)ppollepoll_pwaitnanosleepNo signal is pending on either side.
nanosleepagrees because it does notread the flag, which identifies the flag as the mechanism rather than
anything process-wide.
Why the request can go
The interrupt is not what wakes anybody. The futex waiter wakes on its own
100 ms quantum and re-checks
thread_stop_requested, itswokenflag,expired itimers and queued signals;
futex_interrupt_consumeis one reasonamong several, not the wake.
wakeup_pipe_signalandthread_interrupt_allstay, so threads parked on the shared pipe and inside
hv_vcpu_runstillget the nudge this site sends. Teardown keeps its interrupt: the four
callers that mean it go through
thread_wake_all_blocked, where everythread really is leaving. This site was the only one raising it while the
process carried on running.
The one-shot consume from 520568c is untouched. That fixed the flag staying
set, which is why foot spun on
EINTRforever, and left open whether theflag should have been set at all. The premise dates to the initial import.
Two clean matrix runs bound the liveness risk without closing it. A lost
wake that the interrupt had been masking would surface as a hang rather
than a failure, and the argument above is what rules that out: every
re-check the waiter performs survives the change.
Test
tests/test-futex-no-phantom-eintr.cfails 3 of 7 before the change, passesafter, and passes unchanged on the reference kernel. It is registered in
test-matrix.shunderrun_unit_testsand not intests/manifest.txt, perthat file's scope note, since every assertion is a timeout the guest can
observe.
Its
CLONE_CHILD_CLEARTIDwait is spelled out rather than reusingraw_futex_wait, which carriesFUTEX_PRIVATE_FLAG. The exit-time wake isa plain futex wake, so a private wait never matches its key; the test hung
on the reference kernel until that was a plain wait with a timeout.
Environment and status
MacBookPro18,3, Apple M1 Pro, macOS 26.5.2 (25F84), SDK 26.5, Apple clang
21.0.0. Reference kernel is Linux 6.18.44 through the
qemu-aarch64lane,QEMU 11.1.1. Rebased on
mainat 4388e7c.make checkstops atbuild/test-hellowithaarch64-none-elf-as: No such file or directory; the bare-metal toolchain needs a sudo installer thismachine cannot run, so
test-helloand the fulltest-matrixlanes areunbuilt rather than passing. What did run:
tests/driver.shpasses 98 of98, and
tests/test-matrix.sh elfuse-aarch64passes 280 with 0 failed and10 skipped, twice. That lane is the real-application check, since it runs
busybox, coreutils, dash, lua and jq under the patched binary rather than
unit tests alone. The new test passes 5 of 5 under elfuse and 3 of 3 on the
reference kernel under
QEMU_ACCEL=tcg, where timing is slowest.check-format,check-ascii,check-eintr-contract,check-lock-order,check-atomics,check-skill-refs,check-syscall-coverage,check-svc-tailsand.ci/check-matrix-lists.share clean.