wslc: Switch port relay AcceptThread from WaitForMultipleObjects to IO completion ports#40042
wslc: Switch port relay AcceptThread from WaitForMultipleObjects to IO completion ports#40042benhillis wants to merge 37 commits intofeature/wsl-for-appsfrom
Conversation
* test: enable virtiofs tests and enable WSLG during testing * test fix --------- Co-authored-by: Ben Hillis <benhill@ntdev.microsoft.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Addresses Dependabot alerts #10 and #11. The Microsoft.NETCore.App.Runtime packages (win-x64 and win-arm64) at version 10.0.0 are vulnerable to a denial of service via out-of-bounds read when decoding malformed Base64Url input (CVSS 7.5 High). Bumped to 10.0.4 which includes the fix. Co-authored-by: Ben Hillis <benhill@ntdev.microsoft.com>
Co-authored-by: WSL notice <noreply@microsoft.com>
…ipt (#14424) * Ship initrd.img in MSI using build-time generation via tar.exe Replace the install-time CreateInitrd/RemoveInitrd custom actions with a build-time step that generates initrd.img using the Windows built-in tar.exe (libarchive/bsdtar) and ships it directly in the MSI. The install-time approach had a race condition: wsl.exe could launch before the CreateInitrd custom action completed, causing ERROR_FILE_NOT_FOUND for initrd.img. Changes: - Add CMake custom command to generate initrd.img via tar.exe --format=newc - Add initrd.img as a regular file in the MSI tools component - Remove CreateInitrd/RemoveInitrd custom actions from WiX, DllMain, and wslinstall.def - Remove CreateCpioInitrd helper and its tests (no longer needed) - Update pipeline build targets to build initramfs instead of init * pr feedback * more pr feedback * switch to using a powershell script instead of tar.exe * powershell script feedback * hopefully final pr feedback --------- Co-authored-by: Ben Hillis <benhill@ntdev.microsoft.com>
…uire a call to the service (#14380) * virtiofs: update logic so querying virtiofs mount source does not require a call to the service * more pr feedback * use std::filesystem::read_symlink * pr feedback and use canonical path in virtiofs symlink * make sure canonical path is always used --------- Co-authored-by: Ben Hillis <benhill@ntdev.microsoft.com>
* VirtioProxy: Add IPv6 address, gateway, and route support - Add PreferredIpv6Address field and GetBestGatewayV6* methods to NetworkSettings - Extend GetHostEndpointSettings() to discover IPv6 unicast address and gateway - Add UpdateIpv6Address() using ModifyGuestEndpointSettingRequest<IPAddress> - Push IPv6 default route to guest via UpdateDefaultRoute(AF_INET6) - Remove AF_INET6 early return in ModifyOpenPorts, use INETADDR_PORT() - Add EndpointRoute::DefaultRoute() static factory - Pass client_ip_ipv6 in devicehost options (not yet parsed by devicehost) - Remove gateway_ip from devicehost options (only needed for DHCP) - Include IPv6 DNS servers in non-tunneling DNS settings - Add ConfigurationV6 and DnsResolutionAAAA tests * cleanup and add more ipv6 tests * added test coverage and minor updates * clang format * pr feedback * format source * pr feedback * test fixes --------- Co-authored-by: Ben Hillis <benhill@ntdev.microsoft.com>
* Initial work * . * pr feedback and add unit test * minor tweaks an fix use after free in logging statement * implement PR feedback * hopefully final pr feedback * pr feedback in test function * Address PR feedback: add try/catch to TrackPort and PortZeroBind queue push --------- Co-authored-by: Ben Hillis <benhill@ntdev.microsoft.com>
There were instructions already on how to install tcpdump in WSL, but iptables are also needed for the log collection to be complete, so this PR adds instructions on how to also install iptables. Co-authored-by: Andre Muezerie <andremue@linux.microsoft.com>
Co-authored-by: Ben Hillis <benhill@ntdev.microsoft.com>
* Move all supported Ubuntu images to the new format We backported the build pipeline so all current LTSes come out in the new tar-based format * Remove the appx based distros All WSL users can run tar-based distros by now, right? There is no benefit in maintaining both formats.
- Allow VirtioProxy to keep EnableDnsTunneling=true in config, but clear socket-specific options (BestEffortDnsParsing, DnsTunnelingIpAddress) - Suppress dedicated DNS tunneling hvsocket for VirtioProxy; tunneling is handled through the VirtioNetworking device host instead - Set DnsTunneling flag on VirtioNetworkingFlags so the device host knows to tunnel DNS - Expand SWIOTLB kernel cmdline to cover VirtioFs and VirtioProxy - Bump DeviceHost package to 1.1.39-0 - Add VirtioProxy DNS test coverage for tunneling on/off - Skip GuestPortIsReleasedV6 on Windows 10 Co-authored-by: Ben Hillis <benhill@ntdev.microsoft.com>
Co-authored-by: Ben Hillis <benhill@ntdev.microsoft.com>
* Refactor: trim unnecessary DLL deps from COMMON_LINK_LIBRARIES - Split MSI/Wintrust install functions from wslutil.cpp into install.cpp - Remove MI.lib, wsldeps.lib, msi.lib, Wintrust.lib, computecore.lib, computenetwork.lib, Iphlpapi.lib from COMMON_LINK_LIBRARIES - Add per-target MSI_LINK_LIBRARIES, HCS_LINK_LIBRARIES, SERVICE_LINK_LIBRARIES - Delay-load msi.dll and WINTRUST.dll for wsl.exe and wslg.exe - Result: wslhost, wslrelay, wslcsdk, testplugin lose msi/wintrust startup imports; wsl.exe and wslg.exe defer msi/wintrust loading until actually needed; wslservice is the only target that imports computecore/computenetwork/Iphlpapi * minor fixes to install.cpp that were caught during PR * move to wsl::windows::common::install namespace --------- Co-authored-by: Ben Hillis <benhill@ntdev.microsoft.com>
* detach terminal before running mount -a * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * use _exit on error before execv in child process to avoid unintentional resource release * Add regression test * Fix clang format issue * fix all clang format issue * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * resolve ai comments * move test to unit test * Fix string literal * Overwrite fstab to resolve pipeline missing file issue --------- Co-authored-by: Feng Wang <wangfen@microsoft.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
…pipeline is running on (#14492)
* test: Add arm64 test distro support * update unit test baseline * more test baseline updates --------- Co-authored-by: Ben Hillis <benhill@ntdev.microsoft.com>
* test: remove duplicated DNS test coverage * format source --------- Co-authored-by: Ben Hillis <benhill@ntdev.microsoft.com>
Co-authored-by: WSL localization <noreply@microsoft.com>
…ith DNS over TCP (#14532) Co-authored-by: Ben Hillis <benhill@ntdev.microsoft.com>
…ts (#14531) Co-authored-by: Ben Hillis <benhill@ntdev.microsoft.com>
Co-authored-by: WSL localization <noreply@microsoft.com>
Co-authored-by: WSL localization <noreply@microsoft.com>
* Update cgmanifest to match CMakeLists.txt * Update CMakeLists.txt Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: WSL notice <noreply@microsoft.com>
* Update Microsoft.WSL.DeviceHost to version 1.1.48-0 (#14575) Co-authored-by: Ben Hillis <benhill@ntdev.microsoft.com> * Re-enable WSLG during testing. This reverts commit bf759a0. * add back config change (will work with new default, but makes test explicit) --------- Co-authored-by: Ben Hillis <benhill@ntdev.microsoft.com>
3de1c99 to
3fe1174
Compare
3fe1174 to
8091b71
Compare
8091b71 to
a7ab4d5
Compare
b06a0f2 to
fdb0b87
Compare
…ebugging When /attachdebugger is passed to test.bat, run-tests.ps1 now: - Starts te.exe with /waitfordebugger in the background - Polls for the TE.ProcessHost.exe child process via WMI - Launches WinDbgX attached directly to the test host PID - With /inproc, attaches to TE.exe itself instead This replaces the manual workflow of running /waitfordebugger, reading the PID from the output, and launching WinDbgX separately. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
fdb0b87 to
a9ecbec
Compare
a9ecbec to
677ed96
Compare
677ed96 to
c712ebf
Compare
Per review feedback from @OneBlue: - Add /inproc when /attachdebugger is set so WinDbgX attaches directly to TE.exe instead of polling for TE.ProcessHost.exe - Simplify exit to pass through TE.exe exit code directly Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The previous fix unconditionally called GetOverlappedResult(bWait=TRUE), which hangs in IOCP mode because Overlapped.hEvent is nullptr and cancellation completions go to the IOCP queue, not the file handle. Fix: when RegisteredWithIocp, set a temporary event in the OVERLAPPED before calling CancelIoEx so GetOverlappedResult can wait on it. This ensures the kernel is done with the OVERLAPPED before the destructor frees it, without hanging. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ac96349 to
357a29f
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 90 out of 90 changed files in this pull request and generated 4 comments.
Comments suppressed due to low confidence (1)
NOTICE.txt:645
NOTICE.txtno longer contains any entries forMicrosoft.NETCore.App.Runtime.win-*(and related license text), butpackages.configstill references these runtimes (now updated to 10.0.4). If NOTICE is the authoritative third-party notices file for shipped dependencies, it likely needs to be regenerated/updated to reflect the current set and versions of redistributed components rather than removing these sections entirely.
MIT License
Copyright (c) <year> <copyright holders>
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
---------------------------------------------------------
---------------------------------------------------------
Microsoft.Xaml.Behaviors.WinUI.Managed 3.0.0 - MIT
| void MultiHandleWait::AddHandle(std::unique_ptr<OverlappedIOHandle>&& handle, Flags flags) | ||
| { | ||
| EnsureIocp(); | ||
| handle->Register(m_iocp.get(), handle.get()); | ||
| m_handles.emplace_back(flags, std::move(handle)); |
There was a problem hiding this comment.
MultiHandleWait::AddHandle calls handle->Register(...) and EnsureIocp() uses m_iocp, but OverlappedIOHandle / MultiHandleWait declarations in relay.hpp currently do not declare a Register() method or any m_iocp member. As-is this TU should fail to compile. Please update relay.hpp to match the new IOCP-based design (add the virtual Register(...) API, required members, and update derived handle class declarations accordingly).
| void MultiHandleWait::Cancel() | ||
| { | ||
| m_cancel = true; | ||
| // Wake up GetQueuedCompletionStatus. The key=0 completion will set m_cancel in Run(). | ||
| if (m_iocp) | ||
| { | ||
| PostQueuedCompletionStatus(m_iocp.get(), 0, 0, nullptr); | ||
| } |
There was a problem hiding this comment.
MultiHandleWait::Cancel() no longer sets m_cancel (it only posts a key=0 completion if m_iocp exists). If Cancel() is invoked before Run() has dequeued the posted packet (or before m_iocp is created), the wait loop may continue scheduling work instead of stopping promptly. Consider setting m_cancel = true in Cancel() in addition to posting to the IOCP.
| set(INITRAMFS ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/${CMAKE_BUILD_TYPE}/initrd.img) | ||
| set(INIT ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/${CMAKE_BUILD_TYPE}/init) | ||
| add_custom_command( | ||
| OUTPUT ${INITRAMFS} "${CMAKE_CURRENT_BINARY_DIR}/CmakeFiles/initramfs" | ||
| DEPENDS init ${INIT} | ||
| COMMAND powershell.exe -ExecutionPolicy Bypass -NoProfile -NonInteractive -File "${CMAKE_SOURCE_DIR}/tools/create-initrd.ps1" "${INIT}" "${INITRAMFS}" | ||
| COMMAND ${CMAKE_COMMAND} -E touch "${CMAKE_CURRENT_BINARY_DIR}/CmakeFiles/initramfs" | ||
| VERBATIM) |
There was a problem hiding this comment.
The initramfs custom command unconditionally invokes powershell.exe, which will break configuring/building this target on non-Windows hosts. Also, the marker path uses CmakeFiles (nonstandard casing vs CMakeFiles). Consider guarding this block with if (WIN32) (or using pwsh discovery) and fix the output path casing to avoid brittle build behavior.
| @@ -18,10 +18,10 @@ | |||
| <package id="Microsoft.WSL.bsdtar" version="0.0.2-2" /> | |||
| <package id="Microsoft.WSL.Dependencies.amd64fre" version="10.0.27820.1000-250318-1700.rs-base2-hyp" targetFramework="native" /> | |||
| <package id="Microsoft.WSL.Dependencies.arm64fre" version="10.0.27820.1000-250318-1700.rs-base2-hyp" targetFramework="native" /> | |||
| <package id="Microsoft.WSL.DeviceHost" version="1.1.14-0" /> | |||
| <package id="Microsoft.WSL.DeviceHost" version="1.1.48-0" /> | |||
| <package id="Microsoft.WSL.Kernel" version="6.6.114.1-1" targetFramework="native" /> | |||
| <package id="Microsoft.WSL.LinuxSdk" version="1.20.0" targetFramework="native" /> | |||
| <package id="Microsoft.WSL.TestDistro" version="2.5.7-47" /> | |||
| <package id="Microsoft.WSL.TestDistro" version="2.7.1-1" /> | |||
There was a problem hiding this comment.
The PR description/title focuses on IOCP changes for relay::MultiHandleWait and port relay accept handling, but this PR also includes substantial unrelated updates (e.g., initrd generation/packaging changes, VirtioProxy IPv6 changes/tests, distro metadata updates, .NET runtime/TestDistro/DeviceHost package bumps, NOTICE/cgmanifest updates). If these are intended, please reflect them in the PR description (or consider splitting) so reviewers and release notes capture the full scope.
|
Hey @benhillis 👋 — Following up on this PR. The \wsl-github-pr\ check shows action_required status. There are also 2 unresolved review threads remaining (out of 32 total):
This is a large PR (3170+/1359−, 90 files) — great progress resolving 30 of 32 threads! Just those last 2 items and the CI status to sort out. |
Replace the WaitForMultipleObjects-based wait loop in
relay::MultiHandleWaitwith an IO completion port (IOCP), removing theMAXIMUM_WAIT_OBJECTS(64) handle limit for all callers (IORelay, container IO, socket accept, etc.).Switch the port relay
AcceptThreadfromWaitForMultipleObjectsto a direct IOCP, removing the 63-port mapping limit.relay::MultiHandleWait changes (relay.hpp/cpp)
unique_registered_waitRAII type forRegisterWaitForSingleObjecthandlesRegister()pure virtual toOverlappedIOHandle, implemented by all subclassesRegisterWaitForSingleObjectbridgeRun()to useGetQueuedCompletionStatusinstead ofWaitForMultipleObjectsCancel()posts a key=0 completion to wakeRun(); key=0 setsm_cancelon dequeuem_iocpdeclared beforem_handlesfor correct destructionPort relay changes (localhost.cpp)
CreateIoCompletionPortAcceptExcompletions go straight to the IOCP — no events or thread pool waitsPostQueuedCompletionStatuskey=0AcceptEventfromPortRelay(no longer needed)MAXIMUM_WAIT_OBJECTSport limit check