Skip to content

Stop faking SIGCHLD on a CLONE_THREAD exit - #361

Open
alanhc wants to merge 1 commit into
sysprog21:mainfrom
alanhc:futex-phantom-eintr-v2
Open

Stop faking SIGCHLD on a CLONE_THREAD exit#361
alanhc wants to merge 1 commit into
sysprog21:mainfrom
alanhc:futex-phantom-eintr-v2

Conversation

@alanhc

@alanhc alanhc commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

When the last CLONE_THREAD worker exits, forkipc.c raises the futex
interrupt, which makes the next blocking call in the surviving thread return
EINTR. The comment gave the reason: "In real Linux, child exit delivers
SIGCHLD which interrupts futex_wait with -EINTR."

Linux does not. clone(2) is explicit that a thread created with
CLONE_THREAD sends no signal to its parent when it terminates; the thread
group's exit signal goes to the process's parent, not to a sibling. Nothing
is delivered, so the sibling's wait runs to its timeout.

Reproduction

Clone a CLONE_THREAD worker that exits at once, wait for the
CLONE_CHILD_CLEARTID store, then park for 300 ms with nothing to wake the
caller. tests/test-futex-no-phantom-eintr.c is that program; the three
rows below are what it observes.

call Linux 6.18.44 elfuse before
futex(FUTEX_WAIT) ETIMEDOUT, 303ms EINTR, 101ms
ppoll 0, 303ms EINTR, 0ms
epoll_pwait 0, 311ms EINTR, 302ms
nanosleep 0, 309ms 0, 328ms

No signal is pending on either side. nanosleep agrees because it does not
read the flag, which identifies the flag as the mechanism rather than
anything process-wide.

Why the request can go

The interrupt is not what wakes anybody. The futex waiter wakes on its own
100 ms quantum and re-checks thread_stop_requested, its woken flag,
expired itimers and queued signals; futex_interrupt_consume is one reason
among several, not the wake. wakeup_pipe_signal and thread_interrupt_all
stay, so threads parked on the shared pipe and inside hv_vcpu_run still
get the nudge this site sends. Teardown keeps its interrupt: the four
callers that mean it go through thread_wake_all_blocked, where every
thread really is leaving. This site was the only one raising it while the
process carried on running.

The one-shot consume from 520568c is untouched. That fixed the flag staying
set, which is why foot spun on EINTR forever, and left open whether the
flag should have been set at all. The premise dates to the initial import.

Two clean matrix runs bound the liveness risk without closing it. A lost
wake that the interrupt had been masking would surface as a hang rather
than a failure, and the argument above is what rules that out: every
re-check the waiter performs survives the change.

Test

tests/test-futex-no-phantom-eintr.c fails 3 of 7 before the change, passes
after, and passes unchanged on the reference kernel. It is registered in
test-matrix.sh under run_unit_tests and not in tests/manifest.txt, per
that file's scope note, since every assertion is a timeout the guest can
observe.

Its CLONE_CHILD_CLEARTID wait is spelled out rather than reusing
raw_futex_wait, which carries FUTEX_PRIVATE_FLAG. The exit-time wake is
a plain futex wake, so a private wait never matches its key; the test hung
on the reference kernel until that was a plain wait with a timeout.

Environment and status

MacBookPro18,3, Apple M1 Pro, macOS 26.5.2 (25F84), SDK 26.5, Apple clang
21.0.0. Reference kernel is Linux 6.18.44 through the qemu-aarch64 lane,
QEMU 11.1.1. Rebased on main at 4388e7c.

make check stops at build/test-hello with aarch64-none-elf-as: No such file or directory; the bare-metal toolchain needs a sudo installer this
machine cannot run, so test-hello and the full test-matrix lanes are
unbuilt rather than passing. What did run: tests/driver.sh passes 98 of
98, and tests/test-matrix.sh elfuse-aarch64 passes 280 with 0 failed and
10 skipped, twice. That lane is the real-application check, since it runs
busybox, coreutils, dash, lua and jq under the patched binary rather than
unit tests alone. The new test passes 5 of 5 under elfuse and 3 of 3 on the
reference kernel under QEMU_ACCEL=tcg, where timing is slowest.
check-format, check-ascii, check-eintr-contract, check-lock-order,
check-atomics, check-skill-refs, check-syscall-coverage,
check-svc-tails and .ci/check-matrix-lists.sh are clean.

When the last CLONE_THREAD worker exited, forkipc.c raised the futex
interrupt, which makes the next blocking call in the surviving thread
return EINTR. The comment gave the reason: "In real Linux, child exit
delivers SIGCHLD which interrupts futex_wait with -EINTR."

Linux does not. clone(2) is explicit that a thread created with
CLONE_THREAD sends no signal to its parent when it terminates; the
thread group's exit signal goes to the process's parent, not to a
sibling. Nothing is delivered, so a sibling's wait is not interrupted
and its timeout is what ends it.

Measured on Linux 6.18.44 through the qemu lane, a worker exits and the
main thread then parks for 300 ms with nothing to wake it:

  call                  Linux            elfuse before
  futex(FUTEX_WAIT)     ETIMEDOUT 303ms  EINTR 101ms
  ppoll                 0         303ms  EINTR   0ms
  epoll_pwait           0         311ms  EINTR 302ms
  nanosleep             0         309ms  0     328ms

No signal is pending on either side. nanosleep agrees because it does
not read the flag, which is what identifies the flag as the mechanism
rather than anything process-wide.

Removing the request drops the fabricated errno and nothing else. The
futex waiter is not woken by the interrupt: it wakes on its own 100 ms
quantum and re-checks thread_stop_requested, its woken flag, expired
itimers and queued signals, of which futex_interrupt_consume is one
reason among several. wakeup_pipe_signal and thread_interrupt_all stay,
so threads parked on the shared pipe and inside hv_vcpu_run still get
the nudge this site exists to send.

Teardown keeps its interrupt. All four callers that mean it go through
thread_wake_all_blocked, where every thread really is leaving and the
EINTR is one each of them has to see. This site was the only one that
raised it while the process carried on running.

The one-shot consume in 520568c is untouched. That fixed the flag
staying set, which is why foot spun on EINTR forever; it did not ask
whether the flag should have been set at all.

tests/test-futex-no-phantom-eintr.c parks in each of the three calls
above after a worker exit and fails on an early return. It fails three
of seven before this change, passes after, and passes unchanged on the
reference kernel. It lives in test-matrix.sh's run_unit_tests and not in
tests/manifest.txt, per that file's scope note: every assertion is a
timeout the guest can observe, so it is cross-checkable.

The CLEARTID wait inside it is spelled out rather than reusing
raw_futex_wait, which carries FUTEX_PRIVATE_FLAG. The exit-time wake is
a plain futex wake, so a private wait never matches its key and the test
hung on the reference kernel until it was a plain wait with a timeout.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3 issues found across 4 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="tests/test-futex-no-phantom-eintr.c">

<violation number="1" location="tests/test-futex-no-phantom-eintr.c:89">
P2: When the worker exits while `burn_worker()` is blocked, this reap futex consumes the one-shot phantom EINTR before the calls under test run. Reap without a futex wait, or otherwise preserve the interrupt until the target wait.</violation>
</file>

<file name="src/runtime/futex.c">

<violation number="1" location="src/runtime/futex.c:587">
P3: The comment incorrectly says the one-shot flag gives every teardown thread an EINTR. Document that the atomic interrupt is consumed by one waiter, while teardown state wakes the others.</violation>

<violation number="2" location="src/runtime/futex.c:593">
P3: The file-header comment on futex_interrupt_requested (lines 66-67) is now stale: it still says the flag is 'Used to simulate SIGCHLD delivery when all CLONE_THREAD workers exit', which is exactly the behavior this PR removes and the new futex_interrupt_consume comment explicitly says is gone. Update that header comment to reflect that the interrupt is now raised only by teardown via thread_wake_all_blocked, so the two comments do not contradict each other.</violation>
</file>

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

struct k_timespec quantum = {0, 100L * 1000 * 1000};
long deadline = now_ms() + 5000;
while (__atomic_load_n((volatile int *) &ctid, __ATOMIC_ACQUIRE) != 0) {
raw_syscall6(__NR_futex, (long) &ctid, FUTEX_WAIT, r, (long) &quantum,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: When the worker exits while burn_worker() is blocked, this reap futex consumes the one-shot phantom EINTR before the calls under test run. Reap without a futex wait, or otherwise preserve the interrupt until the target wait.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At tests/test-futex-no-phantom-eintr.c, line 89:

<comment>When the worker exits while `burn_worker()` is blocked, this reap futex consumes the one-shot phantom EINTR before the calls under test run. Reap without a futex wait, or otherwise preserve the interrupt until the target wait.</comment>

<file context>
@@ -0,0 +1,186 @@
+    struct k_timespec quantum = {0, 100L * 1000 * 1000};
+    long deadline = now_ms() + 5000;
+    while (__atomic_load_n((volatile int *) &ctid, __ATOMIC_ACQUIRE) != 0) {
+        raw_syscall6(__NR_futex, (long) &ctid, FUTEX_WAIT, r, (long) &quantum,
+                     0, 0);
+        if (now_ms() > deadline)
</file context>

Comment thread src/runtime/futex.c
Comment on lines +587 to +588
* paths through thread_wake_all_blocked: every thread really is leaving, so the
* EINTR is one each of them has to see. Without the clear, the flag stays set

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The comment incorrectly says the one-shot flag gives every teardown thread an EINTR. Document that the atomic interrupt is consumed by one waiter, while teardown state wakes the others.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At src/runtime/futex.c, line 587:

<comment>The comment incorrectly says the one-shot flag gives every teardown thread an EINTR. Document that the atomic interrupt is consumed by one waiter, while teardown state wakes the others.</comment>

<file context>
@@ -583,13 +583,16 @@ void futex_interrupt_clear(void)
- * does, and the spinning main thread eventually faults in a code path the guest
- * never expects to reach.
+ * clears it, 0 otherwise. The interrupt is a one-shot edge, set by the teardown
+ * paths through thread_wake_all_blocked: every thread really is leaving, so the
+ * EINTR is one each of them has to see. Without the clear, the flag stays set
+ * and every subsequent epoll_pwait, ppoll, futex wait, etc. spins on EINTR
</file context>
Suggested change
* paths through thread_wake_all_blocked: every thread really is leaving, so the
* EINTR is one each of them has to see. Without the clear, the flag stays set
* paths through thread_wake_all_blocked. Teardown state marks every thread as
* leaving; the atomic interrupt itself is consumed by only one waiter.

Comment thread src/runtime/futex.c
* until execve clears it -- in foot's case it never does, and the spinning main
* thread eventually faults in a code path the guest never expects to reach.
*
* forkipc.c set it too, on the last clone-thread exit, for a SIGCHLD that

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The file-header comment on futex_interrupt_requested (lines 66-67) is now stale: it still says the flag is 'Used to simulate SIGCHLD delivery when all CLONE_THREAD workers exit', which is exactly the behavior this PR removes and the new futex_interrupt_consume comment explicitly says is gone. Update that header comment to reflect that the interrupt is now raised only by teardown via thread_wake_all_blocked, so the two comments do not contradict each other.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At src/runtime/futex.c, line 593:

<comment>The file-header comment on futex_interrupt_requested (lines 66-67) is now stale: it still says the flag is 'Used to simulate SIGCHLD delivery when all CLONE_THREAD workers exit', which is exactly the behavior this PR removes and the new futex_interrupt_consume comment explicitly says is gone. Update that header comment to reflect that the interrupt is now raised only by teardown via thread_wake_all_blocked, so the two comments do not contradict each other.</comment>

<file context>
@@ -583,13 +583,16 @@ void futex_interrupt_clear(void)
+ * until execve clears it -- in foot's case it never does, and the spinning main
+ * thread eventually faults in a code path the guest never expects to reach.
+ *
+ * forkipc.c set it too, on the last clone-thread exit, for a SIGCHLD that
+ * clone(2) does not send. That is gone; only a process actually tearing down
+ * fabricates an EINTR now.
</file context>

@jserv jserv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Refine tests/test-futex-no-phantom-eintr.c to be more consistent with existing naming schemes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants