Windows OpenSSH: 'uv trampoline failed to spawn Python child process' (os error 2) / WinError 448 untrusted mount point

[clankerbot99]

Windows OpenSSH + uv/npm reparse-point traversal failure

Audience: maintainers of uv (astral-sh/uv), the Hermes Agent installer/runtime, npm/pnpm, and Win32-OpenSSH.
Goal: explain the precise failure mechanism and concrete upstream fixes so Windows-OpenSSH users stop hitting it.

---

1. Problem statement

When a user runs Hermes Agent (or any tool backed by a uv-managed Python virtualenv, or any Node project with workspace symlinks) over Windows OpenSSH Server, the process fails with one of:

• error: uv trampoline failed to spawn Python child process / Caused by: entity not found (os error 2), or
• OSError: [WinError 448] The path cannot be traversed because it contains an untrusted mount point when the app stat()s a path inside node_modules.

The same command works perfectly in a local PowerShell/cmd session on the same machine.

2. Environment

• Windows 10/11 with the OpenSSH Server Windows Feature enabled.
• Target user is a member of the Administrators group (the norm on single-user dev machines).
• Hermes Agent v0.18.0, installed via the Hermes installer which uses uv (uv = 0.11.14) to provision a managed CPython 3.11 venv.
• uv stores the interpreter at C:\Users\<user>\AppData\Roaming\uv\python\cpython-3.11-windows-x86_64-none, which is a junction pointing at the concrete version dir cpython-3.11.15-windows-x86_64-none.
• The Hermes project uses an npm/pnpm workspace; node_modules contains junctions (hermes-tui → ui-tui, @hermes/ink → ui-tui/packages/hermes-ink, etc.).

3. Root cause (mechanism)

1. OpenSSH on Windows creates the session with a network-type logon token. For accounts that are members of Administrators, Windows applies UAC remote restriction (token filtering): the network token is a filtered token missing several privileges a local interactive logon would carry.
2. Windows reparse points (junctions and symlinks) are subject to a traversal/trust check. Traversing a junction requires the caller's token to be "trusted" for the target. A local interactive token passes; a filtered network token fails with STATUS_UNTRUSTED_MOUNT_POINT (0x80070498 / WinError 448). This is not about ACL/file permissions — it is about the token type/privileges.
3. uv's standalone trampoline (hermes.exe, a ~46 KB uv binary) resolves the interpreter path through the uv\python\... junction. Under the filtered token the spawn of python.exe through the junction fails with os error 2 (the kernel returns the untrusted-mount status, surfaced by Rust std::process as "entity not found").
4. The Hermes chat command does os.stat("node_modules/@hermes/ink/package.json") to detect the TUI. @hermes/ink is an npm workspace junction; the stat fails with WinError 448 under the filtered token.

The two failures share the same underlying cause (filtered network token + reparse point) but surface at different code paths. There is also a third, independent gotcha: because the user is an admin, OpenSSH-Win's default sshd_config (Match Group administrators → AuthorizedKeysFile __PROGRAMDATA__/ssh/administrators_authorized_keys) makes sshd ignore ~/.ssh/authorized_keys, so key auth is refused and the client falls through to a password prompt (on a Mac client this is a password dialog; on Windows-side loopback testing it is a git-bash prompt).

4. Why local works but SSH doesn't (the key insight)

It is not missing environment variables. We dumped the SSH session environment and confirmed APPDATA, LOCALAPPDATA, USERPROFILE, PATH, HOME are all correctly set. The differentiator is purely the logon token type: local interactive (trusted → can traverse reparse points) vs. SSH network (filtered → cannot).

5. User-side fix (what we applied)

Set HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System\LocalAccountTokenFilterPolicy = 1. This disables UAC remote restriction for local admin accounts, so an SSH admin logon receives a full (non-filtered) token that can traverse reparse points. Effect is per new logon (reconnect SSH; no reboot required). After this, both the uv trampoline spawn and the node_modules stat succeed over SSH.

We confirmed the fix is durable: hermes chat -q runs the "Installing TUI dependencies…" step which regenerates the npm junctions, and the full token traverses them without error — so the fix does not need to be re-applied when the app rewrites its node_modules.

(Alternative user-side fix with a stronger security posture: use a dedicated non-admin Windows account for SSH. Non-admin logons are not subject to the filtered-token restriction and traverse reparse points normally. This also sidesteps the administrators_authorized_keys requirement.)

6. Recommendations for upstream fixes

uv (astral-sh/uv) — primary offender for the Python spawn

• Do not install the managed interpreter as a junction. The version-keyed directory (cpython-3.11-windows-x86_64-none) is a junction to the concrete version (cpython-3.11.15-...). On Windows, prefer a real directory copy, or store the resolved absolute path and skip the indirection entirely. (A symlink has the same traversal restriction for filtered network tokens, so it is not a fix.)
• In the standalone trampoline (uv self-contained exe used as a venv launcher), resolve the interpreter path to its final, non-reparse target before CreateProcess. GetFinalPathNameByHandle / DeviceIoControl(FSCTL_GET_REPARSE_POINT) resolve the junction to the real path; spawning through the resolved path avoids the kernel traversal check entirely.
• Surface a better error. When the spawn fails with STATUS_UNTRUSTED_MOUNT_POINT, detect it and emit actionable guidance (e.g., "the Python interpreter is behind a reparse point the current logon token cannot traverse; if connecting over SSH/remote desktop, see …").

Hermes installer / runtime

• Detect the SSH/admin scenario at install time and either (a) provision the managed Python as a real directory rather than a junction, (b) document and optionally offer to set LocalAccountTokenFilterPolicy=1, or (c) recommend a non-admin service account.
• The chat command's TUI detection stat()s a path behind an npm workspace junction. Resolve the real path first (e.g., os.path.realpath / Path.resolve()) before stat-ing, or ship the build without workspace junctions (flatten node_modules). This makes the runtime robust regardless of token type.
• Ship a Windows SSH troubleshooting note covering both the reparse-point/token issue and the administrators_authorized_keys requirement for admin users.

npm / pnpm (workspace symlinks)

• Workspace links are created as junctions on Windows — exactly the reparse points that break under filtered network tokens. There is no easy per-tool fix short of using real copies for the linked packages, but documenting the interaction (and that LocalAccountTokenFilterPolicy=1 or a non-admin account resolves it) would save many users.

OpenSSH-Win (PowerShell/Win32-OpenSSH)

• The Match Group administrators → AuthorizedKeysFile __PROGRAMDATA__/ssh/administrators_authorized_keys default is a well-known footgun. Consider documenting it more prominently, or warning at service start if an admin user's ~/.ssh/authorized_keys is being ignored.

7. Minimal reproduction

1. Windows 10/11, enable OpenSSH Server feature, add an admin user.
2. Install Hermes (uv-managed venv) — creates the uv\python\... junction.
3. ssh adminuser@host "hermes --version" → uv trampoline failed to spawn Python child process.
4. Set LocalAccountTokenFilterPolicy=1, reconnect, repeat → succeeds.

(For diagnosis without a second machine, the same error reproduces via loopback ssh user@localhost using a throwaway ed25519 key placed in administrators_authorized_keys for admin users.)

8. Appendix — commands used in diagnosis/fix

# Make SSH admin logon a full (non-filtered) token — the durable fix
$key = "HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System"
New-ItemProperty -Path $key -Name "LocalAccountTokenFilterPolicy" -Value 1 -PropertyType DWord -Force
(Get-ItemProperty $key).LocalAccountTokenFilterPolicy   # -> 1

# Admin SSH key (SYSTEM-owned file; write elevated)
$pk = (Get-Content "C:\Users\<user>\.ssh\id_ed25519.pub").Trim()
Add-Content -Path "C:\ProgramData\ssh\administrators_authorized_keys" -Value $pk
icacls "C:\ProgramData\ssh\administrators_authorized_keys" /inheritance:r /grant:r "SYSTEM:F" /grant:r "BUILTIN\Administrators:F"

# Authoritative reparse check (PowerShell's -Recurse cache lies; use Python)
#   bool(os.stat(path, follow_symlinks=False).st_file_attributes & 0x400)

Error signatures seen:
  error: uv trampoline failed to spawn Python child process
  Caused by: entity not found (os error 2)
  OSError: [WinError 448] The path cannot be traversed because it contains an untrusted mount point

[GottZ]

This was generated by AI during triage.

Summary

One PR references #61557. #61564 changes only the TUI package-detection probe by resolving the Ink package path before is_file(); the supplied diff shows no change to the uv trampoline or Python child-spawn path, so it demonstrates coverage of only part of the reported symptom set.

Related pull requests

• fix(cli): resolve junctions before TUI ink check for Windows OpenSSH #61564 partial — (+19/-2) — partial fix, keep open for revision: the diff targets WinError 448 during the TUI Ink check, but its OSError fallback retries is_file() on the original unresolved path and may reproduce the same failure. The added broken-symlink test already passes under the prior is_file() behavior and therefore does not verify the intended resolution path; this agrees with the contributor keep_open review on fix(cli): resolve junctions before TUI ink check for Windows OpenSSH #61564.

Suggested consolidation

Do not merge #61564 yet. Keep it open and revise it to make the final existence probe safe when resolve() raises OSError, and add mocked tests that distinguish an unresolved-probe failure from a successful resolved-target probe and cover the resolve-error path; after those changes, it can consolidate the TUI-probe portion of #61557, while any effect on the separately reported trampoline symptom remains unverified by this diff.

Complex graph

flowchart LR
    classDef open fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a
    classDef merged fill:#dcfce7,stroke:#15803d,color:#14532d
    classDef closed fill:#e5e7eb,stroke:#6b7280,color:#1f2937
    classDef unverified fill:#f3f4f6,stroke:#9ca3af,color:#374151
    classDef best stroke-width:3px,stroke:#b45309
    classDef target stroke-width:3px,stroke:#4338ca
    I61557(["issue #61557 (open)"])
    P61564["PR #61564 (open)"]
    P61564 -.->|partial| I61557
    class I61557 open
    class I61557 target
    class P61564 open
    click I61557 "https://github.com/NousResearch/hermes-agent/issues/61557"
    click P61564 "https://github.com/NousResearch/hermes-agent/pull/61564"

Loading

Graph: solid arrow = fixes / best fix, dashed arrow = partial or unverified (see edge label); boxed group = PRs duplicating each other; amber border = best fix; indigo border = target; gray node = closed or no verify verdict yet (state tag in the node label).

Cross-PR triage: Reviewed 1 pull request and 1 issue in this complex. Each diff was read against this issue; Assessment working set: 2 kB of PR diffs, 13 kB of issue/PR text, 1 kB of discussion (1 comments), 1 verify verdict. verdicts reflect diff content, not PR titles. Part of an automated triage batch.