Skip to content

ProcessInteropMessages spins at 100% CPU forever when an interop control socket reaches EOF #41173

Description

@racterub

Windows Version

Microsoft Windows [Version 10.0.26200.8037]

WSL Version

2.7.10.0

Are you using WSL 1 or WSL 2?

  • WSL 2
  • WSL 1

Kernel Version

6.18.33.2-microsoft-standard-WSL2

Distro Version

Ubuntu 24.04.1 LTS

Other Software

Windows Terminal (the affected wsl.exe was the relay for a single long-lived
tab). No third-party software is implicated — the defect is in WSL's own
interop path and is reachable from any terminal.

Related issues

#40651 reports the same symptom — wsl.exe spinning at 100% because a
ReadFile loop has no bytesRead == 0 branch on the synchronous-success path.
It is closed, but not resolved: the microsoft-github-policy-service bot
auto-closed it after seven days of author inactivity, with the note "If you're
still experiencing this issue please re-file it as a new issue."
This report is
that re-file, and it differs in three ways that I think matter:

  1. Different file. wsl.exe relay loops in src/windows/common/relay.cpp miss bytesRead == 0 EOF check on synchronous-success path; cause wsl.exe to spin at 100% on AFD sockets #40651 points at src/windows/common/relay.cpp. A
    maintainer replied
    that ScopedMultiRelay "is indeed missing the read length check. But it's
    only used in set version. So it's likely not the cause of the busy loop you
    are seeing."
    I agree — the loop in this report is
    ProcessInteropMessages in src/windows/common/interop.cpp, which is a
    different function with a different exit structure.
  2. The function is pinned. The original reporter noted "We cannot pin the
    spin to one specific function."
    Here the stack, the thread name, and the
    handle all resolve to a single loop, detailed below.
  3. The requested dumps exist. The same comment asked for "several dumps
    seconds apart with procdump -ma <pid>"
    . I captured three, 20 seconds
    apart, and the zero-byte read is measured in them rather than inferred.

Repro Steps

I do not have a deterministic repro. The trigger is a narrow race, but the
race itself is specific and reviewable, so I've described it precisely rather
than guessing at steps.

What produced it in practice:

  1. Open a WSL2 session in a terminal and leave it open for days
  2. Use it normally — in particular, run Windows binaries from inside Linux
    (explorer.exe ., cmd.exe /c …, git.exe, code, clip.exe, anything
    routed through the WSLInterop binfmt handler)
  3. Watch wsl.exe thread count and CPU over time

Each interop invocation spawns one "Interop" thread. The overwhelming
majority exit normally. Roughly once per 20 hours of ordinary use, one
enters an unterminating loop and never exits. They accumulate; each burns
~100% of one core permanently.

The race that has to be lost. Normal shutdown does not trip this:

  1. The launched Windows process exits → WaitForMultipleObjects wakes on the
    process handle (interop.cpp:467) → loop breaks
  2. Windows sends LX_INIT_PROCESS_EXIT_STATUS (interop.cpp:338)
  3. Linux receives it (binfmt.cpp:320), returns at :398, and its
    wil::unique_fd destructors close all four sockets

Step 1 runs on the async branch, so the happy path works. The bug needs
that order inverted — the Linux side closing Sockets[3] first, in the window
between one Windows-side read completing and the next being issued:

  • Windows already parked in the async wait → EOF arrives via
    GetOverlappedResultBytesRead == 0handled correctly
  • Windows issues ReadFile after the FIN has already landed → completes
    synchronously, success, 0 bytes → spins forever

The losing window is a handful of instructions wide, which is why the rate is
~1 per 20 h rather than one per invocation. SIGINT shares the signal mask
with SIGWINCH (binfmt.cpp:206), so a Ctrl-C'd interop command looks like a
plausible way to get the Linux side closing first.

Forcing it deterministically should be possible by delaying the
Windows-side re-read, or closing the Linux-side control socket on a timer, so
the FIN reliably lands before the next ReadFile.

Expected Behavior

When the interop control socket reaches EOF, ProcessInteropMessages should
break out of its read loop, the "Interop" thread should exit, and wsl.exe
should return to ~0% CPU.

Actual Behavior

The thread never exits. It busy-reads a closed socket at ~100% of one core for
the entire remaining lifetime of the wsl.exe process. Leaked threads
accumulate with no upper bound.

On my machine a single wsl.exe reached 68.5% of a 12-core system (821% of
one core) across 10 leaked threads
, having consumed 849 CPU-hours in 9
days
. Left running it would have saturated all 12 cores. Killing the process
was the only remedy.

The defect

ProcessInteropMessages (src/windows/common/interop.cpp:422) has an
infinite read loop whose body handles only the cases where ReadFile
returns FALSE. When ReadFile completes synchronously and returns TRUE,
the loop takes no action and immediately re-issues the read:

// src/windows/common/interop.cpp:422
DWORD ExitCode = 1;
for (;;)
{
    DWORD BytesRead;
    LX_INIT_WINDOW_SIZE_CHANGED WindowSizeMessage;
    bool Success = ReadFile(MessageHandle, &WindowSizeMessage, sizeof(WindowSizeMessage), &BytesRead, &Overlapped);
    if (!Success)
    {
        const auto LastError = GetLastError();
        if ((LastError == ERROR_BROKEN_PIPE) || (LastError == ERROR_HANDLE_EOF))
        {
            /* ... break ... */
        }

        THROW_LAST_ERROR_IF(LastError != ERROR_IO_PENDING);

        /* async path: waits, then correctly checks (!Success) || (BytesRead == 0) -> break */
    }
    // <- line 480: no `else`. Synchronous success falls through and re-reads.
}

The asymmetry is the bug. The asynchronous completion path correctly
treats BytesRead == 0 as EOF and breaks:

Success = GetOverlappedResult(MessageHandle, &Overlapped, &BytesRead, FALSE);
CancelIo.release();
if ((!Success) || (BytesRead == 0))   // <-- EOF handled here
{
    ...
    break;
}

The synchronous completion path performs no equivalent check.

All three loop exits sit inside if (!Success), so once the socket starts
reporting EOF-as-success the loop is not merely leaky but provably
non-terminating:

Exit Line Requires
ERROR_BROKEN_PIPE / ERROR_HANDLE_EOF 430 Success == FALSE — a socket produces neither
BytesRead == 0 452 Success == FALSE first (async branch)
child process exited 467 Success == FALSE first (async branch)

The third is worth calling out: the WaitForMultipleObjects on
Result->Process is also unreachable, so not even the launched Windows
process exiting can break the loop.

The comment above the loop (line 416) suggests how this happened:

"Read messages from the message handle. Break out of the loop if the pipe
is connection is closed or the process exits."

That is accurate for the WSL1 path, which does pass a real named pipe. The
loop appears to have been reused for the socket path without revisiting EOF
semantics.

Why WSL2 hits this and WSL1 does not

The two call sites pass different kinds of handle:

  • WSL1 / LxBus pathinterop.cpp:559 passes SignalPipe.first.get(),
    a real named pipe (OpenAnonymousPipeNtCreateNamedPipeFile). A closed
    pipe yields ERROR_BROKEN_PIPE, which is handled. No spin.

  • WSL2 / VM pathinterop.cpp:335 passes a socket cast to a handle:

    ExitStatus.ExitCode = ProcessInteropMessages(reinterpret_cast<HANDLE>(Sockets[3].get()), &Result);

    A graceful socket close is EOF-as-success, so neither handled status is
    ever produced, and the loop spins.

Which socket: a Hyper-V socket (AF_HYPERV on Windows, AF_VSOCK on
Linux), created overlapped at hvsocket.cpp:102. Linux opens it first —
binfmt.cpp:153 allocates a fresh listening vsock on an ephemeral port per
interop invocation and passes the port number up in the create-process message
(Message->Port = SocketAddress.svm_port, :161). Windows connects four
times (interop.cpp:283), Linux accepts four (binfmt.cpp:184) — stdin,
stdout, stderr, and the control channel. It is the control channel,
Sockets[3], that ProcessInteropMessages reads, and its peer is the /init
binfmt interpreter inside the utility VM.

This is distinct from the long-lived interop server channel
(UtilConnectToInteropServer(), binfmt.cpp:167), which only carries the
launch request.

Secondary defect (same missing branch)

A window-resize message that completes synchronously is silently discarded —
ResizePseudoConsole is only reached on the asynchronous path. So a resize
delivered inline is dropped rather than applied.

Suggested fix

Handle synchronous completion symmetrically with asynchronous completion:

bool Success = ReadFile(MessageHandle, &WindowSizeMessage, sizeof(WindowSizeMessage), &BytesRead, &Overlapped);
if (!Success)
{
    /* ... unchanged ... */
}
else if (BytesRead == 0)
{
    // Peer closed gracefully. On a socket this surfaces as success with zero
    // bytes rather than ERROR_BROKEN_PIPE.
    if (WI_IsFlagClear(Result->Flags, LX_INIT_CREATE_PROCESS_RESULT_FLAG_GUI_APPLICATION))
    {
        THROW_IF_WIN32_BOOL_FALSE(TerminateProcess(Result->Process.get(), 1));
    }

    break;
}
else
{
    // Synchronous completion with data: currently dropped.
    WI_ASSERT((BytesRead == sizeof(WindowSizeMessage)) && (WindowSizeMessage.Header.MessageType == LxInitMessageWindowSizeChanged));
    const COORD Size{static_cast<SHORT>(WindowSizeMessage.Columns), static_cast<SHORT>(WindowSizeMessage.Rows)};
    THROW_IF_FAILED(ResizePseudoConsole(Result->PseudoConsole.get(), Size));
}

The EOF-handling and resize-handling blocks are now duplicated between the
synchronous and asynchronous paths; factoring each into a small lambda would
be cleaner. Happy to send this as a PR if the approach looks right — I have
not built WSL locally, so it is untested beyond review.

A defensive backstop worth considering separately: these interop threads have
no upper bound and no liveness check, so any future non-terminating condition
in this loop degrades the whole machine silently. A leaked thread is permanent
for the life of the process.

Note on current sources

ProcessInteropMessages is byte-identical between tag 2.7.10 and master
at the time of writing, so this is not fixed in current sources. Release
2.7.11 contains only a MoveDistribution VHD fix and security backports.

Diagnostic Logs

I did not collect collect-wsl-logs.ps1 output — the failure is a CPU spin
with no error path, so nothing is logged. Instead I captured three
full-memory dumps of the affected wsl.exe, 20 seconds apart, and analysed
them in WinDbg.

Identical stack signature (matching hash across all three dumps) on all
ten runaway threads:

ntdll!NtReadFile+0x14
KERNELBASE!ReadFile+0x12a
wsl+0x113c51
wsl+0x111fcf
wsl+0x110bc2
wsl+0x1b35df
kernel32!BaseThreadInitThunk+0x17
ntdll!RtlUserThreadStart+0x2c

All ten threads are named "Interop" — set at interop.cpp:271, inside
the very std::thread that calls ProcessInteropMessages. The four wsl
frames map one-to-one onto the source:

Frame Source
wsl+0x1b35df std::thread trampoline (interop.cpp:268)
wsl+0x110bc2 lambda body (interop.cpp:271–340)
wsl+0x111fcf ProcessInteropMessages (interop.cpp:409)
wsl+0x113c51 the ReadFile call (interop.cpp:426)

Each thread holds a distinct handle, consistent with one leaked thread per
interop session. Windows reports every one as Running with no wait
reason
— they never block, which is the signature of a busy loop rather
than a stalled wait.

The zero-byte read, observed. wsl.exe ships without public symbols, so
BytesRead is not recoverable by name — but ReadFile deposits the byte
count in the OVERLAPPED on the thread stack, which is reachable:

  • ntdll!NtReadFile begins mov r10,rcx and r10 survives the syscall,
    giving the HANDLE directly (0x428, 0x748, 0x710)
  • rdx holds the Event parameter (0x66c); r9 holds ApcContext, which
    KERNELBASE!ReadFile sets to the OVERLAPPED pointer (0xb5c8fff858)
  • reading 32 bytes at that address returns an OVERLAPPED whose hEvent is
    0x66c, matching rdx — confirming the right structure

Internal is the NTSTATUS, InternalHigh the byte count. Across the three
captures of the same thread:

Dump Internal InternalHigh
1 0x103 STATUS_PENDING 0
2 0x103 STATUS_PENDING 0
3 0x0 STATUS_SUCCESS 0

Two samples caught the IRP in flight; the third caught it completed —
success with zero bytes. That is the EOF-as-success case the synchronous
path has no branch for, measured rather than inferred.

One further oddity, offered as an observation rather than a claim:
PointerCount on these three handles ranges 33,764–65,423 and fluctuates
between captures, where ordinary file objects sit in the low tens. I could not
determine the mechanism from a user-mode dump, but it is a consistent
fingerprint of extreme I/O churn on those handles.

Per-thread CPU, showing each thread burning ~85–92% of a core
continuously from the moment it was created, and never exiting:

Thread created CPU hours % of one core since creation
day 0 189.6 ~92%
day 3 125.8 ~90%
day 3 124.0 ~89%
day 4 103.3 ~88%
day 5 85.8 ~89%
day 6 65.9 ~86%
day 6 60.5 ~83%
day 6 59.1 ~83%
day 8 18.7 ~87%
day 8 17.0 ~79%

These sum to the process's entire CPU total, so the loop accounts for 100% of
the consumption.

I still have the three full-memory dumps. I have not attached them because
they are ~72 MB each and, being full-memory captures of a process that relayed
terminal I/O for nine days, may contain environment variables, command lines,
and buffered stdio. Happy to send them to wsl-gh-logs@microsoft.com if
that would help — with private PDBs they should resolve wsl+0x113c51 to an
exact line and let you read BytesRead directly.

Metadata

Metadata

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions