nvbug-6193808: Work around mojibake in nvml.system_get_process_name on WSL

[mdboom]

Summary

cuda.core.system.get_process_name(pid) raises UnicodeDecodeError under
WSL whenever the calling process has a non-C locale (which is the default
state for any CPython process, since the interpreter calls
setlocale(LC_ALL, "") at startup). This is reproducible by running the
cuda_core test suite with any seed that schedules
tests/system/test_system_device.py::test_compute_running_processes before
tests/system/test_system_system.py::test_get_process_name.

The underlying defect is in NVML's WSL implementation (see Root cause
below). This PR adds a scoped, defensive workaround in cuda_core so the
public API returns a correct value on WSL. It also fixes a latent issue
where get_process_name was effectively unusable from a fresh process
because it never primed NVML's per-PID name cache.

Root cause: the WSL mojibake

NVML's nvmlSystemGetProcessName on WSL takes a different code path
depending on the process's current locale. With the default "C" locale,
the function returns the basename portion of /proc/<pid>/exe correctly.
With any other locale (including the typical en_US.UTF-8), it instead
walks an internal UTF-16LE buffer holding the executable path but uses a
4-byte stride (as if the buffer were UTF-32LE). Each "code point" it
pulls is therefore two adjacent ASCII bytes packed into the low and
next-higher bytes of a single 24-bit value. That value is then emitted as
an extended 5-byte UTF-8 sequence (the 0xF8-prefixed encoding used to
represent code points beyond U+10FFFF).

The net result for, say, a Python process whose /proc/<pid>/exe resolves
to:

/home/mdboom/.local/share/uv/python/cpython-3.14.0-linux-x86_64-gnu/bin/python3.14

is that the returned buffer looks like ~180 bytes of f8 … chunks
followed by the correctly-encoded trailing basename, e.g.:

f8 9a 80 80 af  f8 9b 90 81 af  f8 8b b0 81 a5  f8 99 80 81 ad  …  /python3.14\0

Decoding the first chunk illustrates the pattern:

• bytes f8 9a 80 80 af decode as the extended-UTF-8 code point 0x68002F
• that 24-bit value packs 'h' (0x68) in the high byte and '/' (0x2F)
  in the low byte — i.e. the source ASCII bytes /h read as a
  little-endian 32-bit value padded with zeros

Every chunk has this structure; together they spell out the prefix
/home/mdboom/.local/share/uv/python/cpython-3.14.0-linux-x86_64-gnu/bin/
two characters at a time. The trailing /python3.14 is unaffected because
of where the buggy stride leaves the cursor.

Why the workaround needs to "re-prime"

nvmlSystemGetProcessName is cache-driven: the per-PID name is populated
the first time NVML enumerates compute processes that include the PID
(typically via nvmlDeviceGetComputeRunningProcesses_v3). Critically:

• The mojibake is produced during the prime call, not during the read.
• Once a buggy entry is in NVML's cache, switching to the "C" locale and
  re-reading does not unscramble it — the cache survives the locale
  flip.
• Re-running the prime call under the "C" locale overwrites the cached
  entry with the correct UTF-8 string. Subsequent reads (in any locale)
  then return correctly.

So the workaround must do prime + read together under "C".

Behaviour after this PR

• Native Linux / Windows: behaviour is identical to before, except that
  get_process_name now primes the NVML cache automatically. This makes
  it usable from a fresh process (previously a caller had to have
  manually queried device.compute_running_processes first or accept
  NotFoundError).
• WSL: the locale flip is applied around the prime + read sequence,
  so the returned name is the correct UTF-8 string regardless of the
  caller's locale.

Discussion

• Should we instead try to fix this at the cuda_bindings layer and fix it
  for all cuda_bindings users? In that case I guess cuda_core should raise
  an exception from get_process_name if on WSL and the cuda_bindings
  installed is too old?
• I do plan to file the underlying bug with NVML. Locale-sensitive APIs should
  generally be avoided.

[github-actions]

Doc Preview CI
🚀 View preview at https://nvidia.github.io/cuda-python/pr-preview/pr-2118/
https://nvidia.github.io/cuda-python/pr-preview/pr-2118/cuda-core/
https://nvidia.github.io/cuda-python/pr-preview/pr-2118/cuda-bindings/
https://nvidia.github.io/cuda-python/pr-preview/pr-2118/cuda-pathfinder/
Preview will be ready when the GitHub Pages deployment is complete.

[rwgk]

I used Cursor GPT-5.5 1M Extra High

---

GPT Findings

1. Medium: cuda_bindings/docs/source/release/13.2.0-notes.rst and
   cuda_bindings/docs/source/release/13.3.0-notes.rst give an incomplete
   raw-NVML workaround. The PR and nvbug context indicate the corruption happens
   when nvmlDeviceGetComputeRunningProcesses_v3 first primes the PID cache, so
   setting the locale to "C" only before nvml.system_get_process_name can be
   too late if the cache was already populated. The notes should say to prime and
   read under "C", matching the cuda.core workaround.

2. Medium: cuda_core/cuda/core/system/_system.pyx now makes
   get_process_name() depend on successfully enumerating compute processes on
   every device. Any unrelated device-level NVML failure now breaks a per-PID
   lookup, including on non-WSL where this is a behavior change. Consider trying
   the direct read first on non-WSL, or making priming failures narrower.

3. Low: cuda_core/docs/source/release/1.1.0-notes.rst says the WSL workaround
   may hold a global lock, but the implementation uses POSIX per-thread locale
   APIs and no global lock is present. That note looks stale or misleading.

Lightweight Thread-Safety Recommendation

The new locale switching implementation is reasonably thread-safe because it
uses POSIX newlocale/uselocale/freelocale, which scope the "C" locale to
the calling OS thread rather than mutating process-global locale state.

The remaining race is around NVML's process-name cache, which appears to be
process/global driver state. On WSL, another thread could call a cache-priming
path such as nvmlDeviceGetComputeRunningProcesses_v3 under a non-"C" locale
between get_process_name()'s prime and read steps, reintroducing corrupted
cached data.

A lightweight improvement would be to add a module-private Python
threading.RLock shared by the cuda.core paths most likely to touch this
cache. Hold it around:

• system.get_process_name()'s WSL c_locale_guard() + prime + read sequence.
• Device.compute_running_processes on WSL, since that is the main in-package
  path that can prime the process-name cache.

This would not protect raw cuda.bindings.nvml.* calls or external users, but
it would cover the most likely cuda.core race without globally serializing all
NVML access. A Python RLock is sufficient here: even if Cython releases the GIL
inside NVML wrappers, the lock remains held until the surrounding Python with
block exits.

[kkraus14]

Why is this in both 13.2 and 13.3 release notes?

[rwgk]

It's a compatibility note that's in line with the existing v12.6.2.post1 and below note, which we've been carrying forward since 12.8.0, I believe in every single release since then:

$ git grep 'v12.6.2.post1 and below' | cutniq | cut -d/ -f-4 | uniq -c
     16 cuda_bindings/docs/source/release
     15 cuda_python/docs/source/release

[kkraus14]

There is already a WSL test helper with broader detection logic: cuda_python_test_helpers._detect_wsl checks /proc/sys/kernel/osrelease and also falls back to WSL_DISTRO_NAME / WSL_INTEROP. Because the test helpers are used by cuda.bindings tests and should not depend on cuda.core, we unfortunately probably need to keep this small functionality duplicated. Could we pull the env-var fallback into this runtime helper too, so the two helpers have the same detection coverage?

[rwgk]

I looked at the git history, cuda_python_test_helpers._detect_wsl was added in #1045. That PR introduced the WSL_DISTRO_NAME / WSL_INTEROP. To me it looks like purely defensive AI-generated code that I often saw in the older agents; even in newer ones I still sometimes need to clean out defensive code. I quizzed GPT-5.5 about it; it told me:

So yes: those env vars look mostly defensive. ... env vars can be inherited into containers/subprocesses, so the fallback is slightly more false-positive-prone than the /proc check.

I'd argue for removing the fallback, although definitely not in this PR.

[kkraus14]

Fair enough. We should just keep the logic consistent if possible. Ideally we'd only have one implementation, but because we want the test helpers to not depend on cuda-core and we don't want to ship this as part of cuda-bindings / cuda-pathfinder I don't think it's worth a new utilities package or anything of the like.