EXPERIMENT — NOT FOR MERGING — cuda-pathfinder in build-system.requires

[rwgk]

Based on #1801, with commit f44a308 from 1801 reverted here.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[rwgk]

/ok to test

[rwgk]

Tracking Cursor Opus 4.6 1M Thinking analysis that was the motivation for this experimental PR:

My prompt:

Could you please help me develop my understanding: assume cuda-pathfinder 1.5.0 (this PR) is released on PyPI. Could we then revert the last commit? Or will
this still be an issue:

> the cuda namespace package from backend-path=["."] shadows the installed cuda-pathfinder

Cursor response:

 The namespace issue would persist even with 1.5.0 on PyPI. The problem isn't about which version is available — it's structural.
  Here's the mechanism: pyproject-build creates an isolated venv, installs build-system.requires (which would include cuda-pathfinder), then runs the build backend
  in a subprocess. In that subprocess, backend-path=["."] prepends cuda_bindings/ to sys.path. Now sys.path has two locations providing the cuda namespace:
  • cuda_bindings/cuda/ (from backend-path, contains bindings/)
  • <venv>/site-packages/cuda/ (from installed cuda-pathfinder, contains pathfinder/)

  Both are implicit namespace packages (no __init__.py). In theory, Python should merge them into a single namespace with __path__ covering both locations. In
  practice, this failed in CI — and cuda-pathfinder 1.4.3 IS on PyPI and IS a pure-Python wheel, so it should have installed fine. The namespace structure in 1.5.0
  would be identical.
  I'll caveat: I'm not 100% certain the root cause is namespace shadowing vs. a silent pip install failure inside the cibuildwheel container. You could verify by
  adding a diagnostic like CIBW_BEFORE_BUILD: "pip list | grep cuda" to confirm whether the package is actually installed. But either way, the os.environ.get()
  approach is the right call for build backends — it has zero external dependencies and the full get_cuda_path_or_home() features (symlink comparison, warnings,
  caching) aren't useful during a build.

[rwgk]

Tracking Cursor Opus 4.6 1M Thinking analysis of https://github.com/NVIDIA/cuda-python/actions/runs/23381904071 logs (at commit 47ea03e here):

  Mystery solved definitively:
  • Line 5128: `Successfully installed cuda-pathfinder-1.4.3` — pip installed it successfully into the isolated venv at /tmp/build-env-iic2uvbl/
  • Line 5147: The _in_process.py subprocess runs from /opt/python/cp312-cp312/lib/python3.12/site-packages/ — that's the host Python's site-packages, not the
    isolated venv
  • Line 5162: ModuleNotFoundError: No module named 'cuda.pathfinder'

  This confirms the namespace package resolution issue. cuda-pathfinder 1.4.3 is installed and present, but from cuda.pathfinder import ... fails because
  backend-path=["."] puts /project/cuda_bindings/ on sys.path — its cuda/ directory (containing bindings/) prevents the namespace from merging with the venv's
  cuda/pathfinder/.
  Releasing 1.5.0 on PyPI would not help. The issue is structural: Python's namespace package resolution doesn't merge cuda/ across the backend-path and the
  isolated venv's site-packages in this environment. The os.environ.get() approach on CUDA_PATH_CUDA_HOME_cleanup is the right fix.

[rwgk]

/ok to test

[github-actions]

Doc Preview CI
🚀 View preview at https://nvidia.github.io/cuda-python/pr-preview/pr-1803/
https://nvidia.github.io/cuda-python/pr-preview/pr-1803/cuda-core/
https://nvidia.github.io/cuda-python/pr-preview/pr-1803/cuda-bindings/
https://nvidia.github.io/cuda-python/pr-preview/pr-1803/cuda-pathfinder/
Preview will be ready when the GitHub Pages deployment is complete.

[rwgk]

The below is based on analyzing the 3 successful build logs and corresponding 9 successful tests logs from this CI run:

https://github.com/NVIDIA/cuda-python/actions/runs/23382608485?pr=1803

---

Build hooks patch: use cuda.pathfinder at build time

Apply after cuda-pathfinder >= 1.5 is released on PyPI and the
build-system.requires in both cuda_bindings/pyproject.toml and
cuda_core/pyproject.toml pin cuda-pathfinder >= 1.5.

Root cause (confirmed by CI diagnostic on 3 platforms)

When pyproject-build (or pip) builds a wheel it:

1. Creates an isolated venv and installs build-system.requires
   (including cuda-pathfinder).
2. Prepends the project directory to sys.path (via backend-path=["."])
   and imports the build backend (build_hooks).
3. Removes the project directory from sys.path again (via the
   _BackendHook context manager inside pyproject_hooks).

Between steps 2 and 3 the cuda namespace package has already been
resolved. Python caches it in sys.modules with a _NamespacePath that
points only to the project's cuda/ directory
(e.g. /project/cuda_bindings/cuda). The site-packages cuda/ directory
(which contains pathfinder/) is never merged in.

Key evidence from all 6 diagnostic blocks (2 packages × 3 platforms):

cuda.__path__: _NamespacePath(['/project/cuda_bindings/cuda'])
cuda.__file__: None
import cuda.pathfinder failed (before extend_path): No module named 'cuda.pathfinder'
cuda.__path__ (after extend_path): _NamespacePath(['/project/cuda_bindings/cuda'])
import cuda.pathfinder failed (after extend_path): No module named 'cuda.pathfinder'

pkgutil.extend_path did not add the site-packages path. This is
because _NamespacePath._recalculate() is driven by the path finders
(not os.path.isdir), and the import machinery has already locked the
cuda namespace to a single portion.

Fix strategy

Replace cuda.__path__ (a _NamespacePath) with a plain list that
includes both the project's cuda/ and the site-packages cuda/.
A plain list is not subject to _NamespacePath._recalculate(), so the
additional path sticks.

Patch

Apply to both cuda_bindings/build_hooks.py and cuda_core/build_hooks.py.

Step 1 — add _import_get_cuda_path_or_home helper

def _import_get_cuda_path_or_home():
    """Import get_cuda_path_or_home, working around PEP 517 namespace shadowing.

    In isolated build environments, backend-path=["."] causes the ``cuda``
    namespace package to resolve to only the project's ``cuda/`` directory,
    hiding ``cuda.pathfinder`` installed in the build-env's site-packages.
    Fix by replacing ``cuda.__path__`` with a plain list that includes the
    site-packages ``cuda/`` directory.
    """
    try:
        import cuda.pathfinder
    except ModuleNotFoundError:
        pass
    else:
        return cuda.pathfinder.get_cuda_path_or_home

    # In PEP 517 isolated builds, the cuda namespace resolves to only the
    # project's cuda/ dir (via backend-path=["."]).  The _NamespacePath
    # cached in sys.modules never merges the site-packages cuda/ where
    # pathfinder lives.  Replace it with a plain list so the fix sticks.
    import cuda

    for p in sys.path:
        sp_cuda = os.path.join(p, "cuda")
        if os.path.isdir(os.path.join(sp_cuda, "pathfinder")):
            cuda.__path__ = list(cuda.__path__) + [sp_cuda]
            break

    import cuda.pathfinder

    return cuda.pathfinder.get_cuda_path_or_home

Step 2 — replace _get_cuda_path

@functools.cache
def _get_cuda_path() -> str:
    get_cuda_path_or_home = _import_get_cuda_path_or_home()
    cuda_path = get_cuda_path_or_home()
    if not cuda_path:
        raise RuntimeError("Environment variable CUDA_PATH or CUDA_HOME is not set")
    print("CUDA path:", cuda_path)
    return cuda_path

Step 3 — remove _diagnose_namespace_packages

Delete the _diagnose_namespace_packages() function entirely.

Step 4 — restore cuda-pathfinder in build-system.requires

In both cuda_bindings/pyproject.toml and cuda_core/pyproject.toml,
add "cuda-pathfinder>=1.5" back to the [build-system] requires list.

[rwgk]

/ok to test

[rwgk]

/ok to test