Skip to content

VcXsrv WSL2 vsock Fix

millin edited this page Jul 22, 2026 · 2 revisions

VcXsrv WSL2 vsock Fix

Objective: Make the (already compiled-in but effectively unusable) Hyper-V vsock transport work out of the box for WSL2: automatic unprivileged VM-id detection, self-healing rebind across WSL restarts, and a display-number-aware port convention — with zero changes required from WSL2 users beyond a single socat relay command.

Date: 2026-07-23
Upstream: https://github.com/marchaesen/vcxsrv
Issue: #80 | PR: TBD
Dev Branch: https://github.com/vivimillin/vcxsrv/tree/feature/vsock-wsl2

Commits:

  • 0d17e2d5e xtrans: fix hyperv listener ignoring the display number
  • edc667d1e hw/xwin: add -vsock option for zero-config WSL2 Hyper-V socket support
  • 7620fac16 hw/xwin: watch the WSL2 VM instance instead of polling the VM id

1. Problem Analysis

1.1 Symptoms

Running X clients from WSL2 against VcXsrv over the default TCP path (NAT + localhost forwarding):

Pain point Effect
Latency through the WSL2 localhost relay Visible stutter with mouse-heavy event streams (SDL2 games, drag operations)
VPN on/off, sleep/resume Connections may drop or the host IP changes
Windows Firewall Prompts for inbound allowance on some setups
Configuration Users must look up the host IP or enable localhostForwarding; DISPLAY is non-obvious

Hyper-V vsock bypasses the network stack entirely and is the ideal transport for this — and VcXsrv has in fact shipped one for years. It just did not work for WSL2.

1.2 What Already Existed

The hyperv xtrans transport arrived via routine upstream sync (libxtrans ~2020, motivated by WSL scenarios) and is compiled in (include/dix-config.hHYPERV 1):

Piece Location Status
SocketHyperV{CreateListener,Accept,GetAddr,GetPeerAddr} X11/xtrans/Xtranssock.c Working
Transport registration + IsLocal(AF_HYPERV) X11/xtrans/Xtrans.c Working
ConvertAddr: AF_HYPERV → FamilyLocal (no cookie needed) xorg-server/os/access.c Working
-vmid {GUID} / -vsockport options xorg-server/os/utils.c Working
-listen/-nolisten hyperv generic xtrans mechanism Working

Live test against the installed vcxsrv confirmed the transport is sound: vcxsrv.exe :11 -vmid {wsl2-vm-guid} + socket(AF_VSOCK) from WSL2 to cid 2, port 106000 → full X11 setup handshake SUCCESS, no cookie.

1.3 Problem 1: WSL2 Never Matches a Wildcard-Bound Listener

The listener's bind address contains a VmId field selecting which VM may connect. The default, HV_GUID_WILDCARD, works for regular Hyper-V VMs — but WSL2's utility VM only connects to listeners bound to its exact VM id (microsoft/WSL#5751; X410 users hit the same wall in microsoft/WSL#4723).

So -vmid {guid} is required — and the id changes at every WSL VM boot (wsl --shutdown, Windows reboot). The documented discovery methods are elevated: hcsdiag list (admin), or the HCS APIs used by X410 ("Hyper-V Administrators" group).

Net effect: the feature was effectively unusable for its primary audience.

1.4 Problem 2: Display Number Ignored on Bind

SocketHyperVCreateListener() bound ServiceId.Data1 = vsockPort (fixed 106000) regardless of the display number, so vcxsrv :0 and vcxsrv :1 collided on the same vsock port. The client-side code in the same file already used 106000 + display — the server side was simply never finished.

1.5 Problem 3: Listener On by Default

The hyperv listener was created on every machine by default:

  • Hosts without Hyper-V got confusing startup errors (see upstream issue #53's log).
  • AF_HYPERV peers are treated as FamilyLocal, so any local VM that could reach the listener was admitted without MIT-MAGIC-COOKIE.

2. Fix Principles

Keep the generic transport generic; add a small WSL2 convenience layer on top, all opt-in via a single -vsock flag:

WSL2 (user runs one socat)                 Windows host
+------------------------+                +-------------------------------+
| X app  DISPLAY=:0      |                | vcxsrv.exe -vsock             |
|  | unix /tmp/.X11-unix/X0               |  OsVendorPreInit:             |
|  v                     |   AF_VSOCK     |   wslinfo --vm-id -> bind     |
| socat ... VSOCK-CONNECT:2:106000 <----> |  xtrans hyperv listener       |
+------------------------+  cid 2 = host  |  VmId={VM GUID}               |
   (guest never needs      port 106000+n  |  ServiceId={106000+display}-  |
    any GUID)                             |   facb-11e6-bd58-...          |
                                          |  worker thread: watch VM      |
                                          |  instance -> OsTimer: rebind  |
                                          +-------------------------------+

Three commits, three separable layers:

  1. xtrans bug fix (display offset): :0 keeps 106000, :1 gets 106001
  2. -vsock auto-configuration: preinit binds the current WSL2 VM id, unprivileged; default (no -vsock/-vmid/-listen hyperv) is now not to listen at all
  3. Self-healing watcher: worker thread watches the VM instance (free), main-thread timer rebinds only when it changed

3. Modified File Summary

File Changes
X11/xtrans/Xtranssock.c SocketHyperVCreateListener: bind vsockPort + display
xorg-server/hw/xwin/winvsock.c New. winVsockPreInit + watcher (worker + OsTimer)
xorg-server/os/connection.c HyperVRebindListener() (WIN32+HYPERV only): tear down + recreate the hyperv listener at runtime
xorg-server/hw/xwin/InitOutput.c Call sites (OsVendorPreInit, end of InitOutput) + -help text
xorg-server/hw/xwin/winprocarg.c -vsock / -novsock parsing
xorg-server/hw/xwin/winglobals.{c,h} Bool g_fVsock = FALSE;
xorg-server/hw/xwin/win.h Two prototypes
xorg-server/hw/xwin/makefile, meson.build Build wiring
xorg-server/hw/xwin/man/XWin.man Option + "WSL2 AND HYPER-V VSOCK" section

4. Step One: xtrans Display Offset

SocketHyperVCreateListener(), X11/xtrans/Xtranssock.c:

	/*
	 * The port argument is really the display number string. Add it to the
	 * base port so that concurrent server instances (e.g. :0 and :1) bind
	 * distinct service ids (106000, 106001, ...), matching the client-side
	 * convention in SocketHyperVConnect(). Previously the display number was
	 * ignored here and every instance collided on the same port.
	 */
	sockname.ServiceId.Data1 = vsockPort +
	    (port ? (unsigned int) strtoul (port, (char **)NULL, 10) : 0);

:0 keeps using 106000, so existing recipes (the maintainer's SourceForge thread, community gists) are unaffected. The base port stays 106000 deliberately; -vsockport overrides it.


5. Step Two: -vsock Feature

5.1 Startup Path (OsVendorPreInit, before listeners are created)

winVsockPreInit(argc, argv) in winvsock.c:

  1. Not -vsock: unless the user passed -vmid or -listen hyperv, call _XSERVTransNoListen("hyperv") (closes the default wildcard listener) and return.
  2. -vsock with -vmid: manual mode wins; log and return.
  3. -vsock: if a vmmemWSL process exists, query the VM id via wsl.exe -- wslinfo --vm-id (captured through a pipe, CREATE_NO_WINDOW, 5 s timeout), validate the 36-char GUID strictly, then _XSERVTransSetHyperVVmId("{guid}").
  4. Any failure → _XSERVTransNoListen("hyperv") + one ErrorF line; TCP is never affected. The watcher still starts, so the listener appears automatically once a WSL2 VM starts.

Ordering note: dix/main.c runs OsVendorPreInit (line ~195) before CreateWellKnownSockets (line ~211), so a VM id set here is picked up by the initial bind.

Canonical startup: vcxsrv.exe :0 -multiwindow -clipboard -wgl -vsock. XLaunch users can equivalently set ExtraParams="-vsock" in config.xlaunch (the wizard's "Additional parameters for VcXsrv" box); the shortcut stays xlaunch.exe -run config.xlaunch.

5.2 Runtime Rebind (os/connection.c)

xtrans has no usable ResetListener for the socket family, so HyperVRebindListener() (guarded #if defined(WIN32) && defined(HYPERV)) does a manual teardown + recreate on the main thread:

    /* Tear down an existing hyperv listener, if any */
    for (i = 0; i < ListenTransCount; i++) {
        if (... TransName != "hyperv") continue;
        fd = _XSERVTransGetConnectionNumber(ListenTransConns[i]);
        RemoveNotifyFd(fd);
        _XSERVTransClose(ListenTransConns[i]);
        /* compact the arrays */
    }

    _XSERVTransListen("hyperv");          /* clear our own startup nolisten */
    snprintf(name, sizeof(name), "hyperv/:%s", display);
    conn = _XSERVTransOpenCOTSServer(name);
    _XSERVTransCreateListener(conn, display, 0);
    /* append to ListenTransConns/Fds, SetNotifyFd(EstablishNewConnections_local) */

It doubles as the late-create path: when no hyperv listener exists (WSL2 was not running at startup), the same function creates it from scratch.

5.3 Watcher (worker thread + OsTimer)

The worker thread (clipboard-style pthread, detached) never calls into the X server; it only posts a new VM id into a small CRITICAL_SECTION-guarded state block. The main-thread OsTimer callback (re-arming every tick) applies it:

    if (doRebind) {
        snprintf(braced, sizeof(braced), "{%s}", guid);
        _XSERVTransSetHyperVVmId(braced);
        if (HyperVRebindListener() == 0) { /* record bound guid, ErrorF */ }
        /* on failure the dirty flag stays set: retried next tick */
    }

The watcher is started at the end of InitOutput (gated on serverGeneration == 1), because OsInit() (which calls TimerInit()) runs after OsVendorPreInit.


6. Step Three: Watcher Optimization

First version queried the VM id — i.e. spawned a wsl.exe process — on every 2 s poll while the VM was running. The optimization tracks the vmmemWSL process instead:

typedef struct {
    DWORD pid;
    ULONGLONG creationTime;     /* 0 when unavailable */
} winWslVmInfo;

Per poll (~0.5 s, pure in-process API calls):

        if (!winWslVmInstance(&curVm)) { lastVm = {0,0}; continue; }

        if (curVm.creationTime != 0 &&
            curVm.pid == lastVm.pid &&
            curVm.creationTime == lastVm.creationTime)
            continue;           /* same VM instance: nothing to do */

        if (!winWslQueryVmId(guid))     /* VM instance changed (or first
                                           poll / data unavailable) */
            continue;                   /* still booting: retry next poll */
        lastVm = curVm;
        /* compare with bound guid, post dirty flag if different */

Result: while the VM runs unchanged, zero process spawns; when the VM restarts, the new instance (different pid/creation time) triggers exactly one wsl.exe -- wslinfo --vm-id. The lower cost allowed shortening poll and timer intervals from 2 s/1 s to 0.5 s/0.5 s — the listener now follows a WSL restart in ~1.5 s worst case.


7. Technical Notes

7.1 Why wsl.exe -- wslinfo --vm-id and Not HCS APIs

X410 solves the same problem via the Host Compute Service APIs, which require "Hyper-V Administrators" group membership (and needed a reliability fix in their 3.2.1). wslinfo --vm-id (Store WSL 2.0+) is fully unprivileged and self-validating: it runs inside the VM, so a successful reply proves the VM is alive. Fallbacks (old WSL, WSL1, no WSL) all fail the strict GUID validation and degrade silently to TCP.

7.2 Why the vmmemWSL Process Gate Matters

wsl.exe -- <cmd> boots the WSL VM if it is stopped. A naive wslinfo poll would resurrect the VM every few seconds after the user ran wsl --shutdown. The vmmemWSL ("vmmem" on older Windows) process check never spawns anything and never boots the VM; wsl.exe is only spawned when the VM is already up.

7.3 Why Worker Thread + OsTimer Instead of Synchronous Polling

Each VM-id query costs ~0.1–0.5 s (process spawn). Doing that in the OsTimer callback would freeze the X server's main loop — all clients, including mouse input, would stutter every poll. The split design keeps the main thread to a flag check (nanoseconds); spawning and waiting happen on the worker. The rebind itself (close + socket + bind + listen) is microseconds.

7.4 Why Pid + Creation Time

Pids are recycled; a new vmmemWSL could reuse an old pid. The creation time (GetProcessTimes) disambiguates. When OpenProcess fails (e.g. permissions), creationTime stays 0 and the code degrades to querying the VM id every poll — same behavior as the unoptimized version.

7.5 Port Convention: Why Keep 106000

The maintainer documented vcxsrv :0 -vmid {GUID} -vsockport 106000 on SourceForge (chosen deliberately: vsock ports are 32-bit, not limited to 65535), and community recipes use it. Keeping 106000 + display means zero breakage; -vsockport 6000 is available for X410-style parity.

7.6 Why Default Off

Default-on wildcard listening was a security surface (any local VM treated as FamilyLocal, no cookie) and produced log noise on non-Hyper-V hosts. Default is now silent; -vsock / -vmid / -listen hyperv are the three explicit ways in.

7.7 Trust Model of vsock Clients

vsock connections map to FamilyLocal (same trust as 127.0.0.1) and skip cookie authorization by default. Only the single bound VM can reach the listener — no wildcard exposure.

7.8 socat retry Semantics (the race that is not one)

socat's VSOCK-CONNECT runs per accepted client in a forked child — never at startup. With forever,retry=10,interval=2, a client connecting during the ~1.5–3.5 s rebind gap simply waits while the child retries (~20 s window); the listener itself never dies. So the documented relay line makes the WSL-restart race practically invisible to users.

7.9 Multi-Display, Multi-VM Matrix

Scenario Result
Many X clients, one server, one port Native listen/accept fan-out, always worked
Two servers (:0, :1) 106000/106001 after the offset fix; second instance should use -noclipboard
WSL2 + regular VM simultaneously Not possible with one listener (single VmId); workaround: two instances (-vsock + -listen hyperv)
WSL1 No VM layer; -vsock degrades silently to TCP

8. Verification

No test suite exists in this build; verification was end-to-end on Windows 11 + Store WSL2 (Ubuntu):

# Test Result
1 vcxsrv :11 -vsock with WSL2 running Log shows winVsock: WSL2 VM <guid> detected, vsock listener enabled
2 socat UNIX-LISTEN:X11 → VSOCK-CONNECT:2:106011 + X11 handshake SUCCESS (status 1)
3 DISPLAY=:11 xeyes Renders; ss --vsock shows ESTAB … 2:106011 while running
4 Two instances :11/:12, both -vsock 106011/106012 both connectable (offset fix)
5 Default (no flags) vsock connect times out; TCP 6012 LISTENING (regression-free)
6 WSL VM restarted while servers kept running Both instances logged vsock listener rebound to WSL2 VM <new-guid>; clients reconnected successfully
7 Post-optimization smoke test Build clean (0 warnings); bind + xeyes + ESTAB all pass

Useful verification one-liners:

ss --vsock | grep '2:106000'      # live vsock connections (run while app is up)
vcxsrv.exe :0 -vsock -nolisten tcp -nolisten inet -nolisten inet6
                                  # elimination proof: if X works, it must be vsock

Clone this wiki locally