Skip to content

WIP: pathfinder_compatibility_guard_rails#1977

Draft
rwgk wants to merge 10 commits intoNVIDIA:mainfrom
rwgk:pathfinder_compatibility_guard_rails
Draft

WIP: pathfinder_compatibility_guard_rails#1977
rwgk wants to merge 10 commits intoNVIDIA:mainfrom
rwgk:pathfinder_compatibility_guard_rails

Conversation

@rwgk
Copy link
Copy Markdown
Contributor

@rwgk rwgk commented Apr 25, 2026

Resolves #1038

Continuation of #1936

WIP — CI testing

rwgk added 6 commits April 24, 2026 15:15
Introduce CompatibilityGuardRails plus related errors and tests so callers can opt into CTK and driver compatibility checks while reusing the existing pathfinder lookup APIs.

Made-with: Cursor
Expose process_wide_compatibility_guard_rails at import time so follow-up changes can route the default cuda.pathfinder APIs through a stable public instance. Document the singleton and pin its public availability with a small regression test.

Made-with: Cursor
Make the process-wide CompatibilityGuardRails instance the default path for the public load/find/locate APIs so top-level calls share compatibility state. Factor the routing/fallback/cache-reset glue into a dedicated internal module to keep `cuda.pathfinder.__init__` focused on the public surface, and fall back to the existing raw resolvers when v1 guard rails only have insufficient metadata.

Made-with: Cursor
Allow CUDA_PATHFINDER_COMPATIBILITY_GUARD_RAILS to select strict, best_effort, or off behavior so we can experiment with stricter compatibility checks without changing the public API shape.

Made-with: Cursor
Treat driver-packaged libraries as compatibility-neutral so strict mode can load NVML and other driver libs without a raw fallback, while CTK-backed artifacts remain the only items that establish and enforce the process-wide CTK anchor.

Made-with: Cursor
Infer the CUDA Toolkit line from both wildcard-pinned and range-based cuda-toolkit requirements so strict process-wide guard rails keep working for editable wheel installs used by nvrtc and nvJitLink.

Made-with: Cursor
@copy-pr-bot
Copy link
Copy Markdown
Contributor

copy-pr-bot Bot commented Apr 25, 2026

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@rwgk rwgk self-assigned this Apr 25, 2026
@github-actions github-actions Bot added the cuda.pathfinder Everything related to the cuda.pathfinder module label Apr 25, 2026
@rwgk rwgk added the P0 High priority - Must do! label Apr 25, 2026
@rwgk rwgk added this to the cuda.pathfinder next milestone Apr 25, 2026
@rwgk
Copy link
Copy Markdown
Contributor Author

rwgk commented Apr 25, 2026

/ok to test

@github-actions
Copy link
Copy Markdown

@rwgk rwgk added the feature New feature or request label Apr 25, 2026
@rwgk
Copy link
Copy Markdown
Contributor Author

rwgk commented Apr 25, 2026

Analysis of CI failures for workflow run

Cursor GPT-5.4 Extra High Fast


Findings

  • The failures are not universal. All 34 failing Test_*.txt files I found are local matrix jobs; I did not find failing wheels jobs in this archive.
  • There is one real failure family: strict process-wide guard rails reject CTK dynamic libraries loaded from the checked-out local toolkit tree because pathfinder cannot infer a CTK line for those paths. Representative Linux log: /wrk/logs_24924024509/Test_linux-64___py3.11__13.0.2__local__l4.txt:7237, /wrk/logs_24924024509/Test_linux-64___py3.11__13.0.2__local__l4.txt:7307, /wrk/logs_24924024509/Test_linux-64___py3.11__13.0.2__local__l4.txt:7369, /wrk/logs_24924024509/Test_linux-64___py3.11__13.0.2__local__l4.txt:7442. Representative Windows log: /wrk/logs_24924024509/Test_win-64___py3.10__13.0.2__local__rtxpro6000__TCC_.txt:10323, /wrk/logs_24924024509/Test_win-64___py3.10__13.0.2__local__rtxpro6000__TCC_.txt:10387, /wrk/logs_24924024509/Test_win-64___py3.10__13.0.2__local__rtxpro6000__TCC_.txt:10443, /wrk/logs_24924024509/Test_win-64___py3.10__13.0.2__local__rtxpro6000__TCC_.txt:10510.
  • The 12.9 local jobs hit the same root cause later and therefore fail in cuda_core rather than cuda_bindings, because nvJitLink is the first affected library touched there. Examples: /wrk/logs_24924024509/Test_linux-64___py3.10__12.9.1__local__v100.txt:7389 and /wrk/logs_24924024509/Test_win-64___py3.13__12.9.1__local__l4__TCC_.txt:10289.
  • This matches current intended behavior exactly. In the same failing jobs, tests/test_compatibility_guard_rails.py::test_missing_version_json_raises_insufficient_metadata passes, for example /wrk/logs_24924024509/Test_linux-64___py3.11__13.0.2__local__l4.txt:6414 and /wrk/logs_24924024509/Test_win-64___py3.10__13.0.2__local__rtxpro6000__TCC_.txt:8971.
  • The ##[error]WARNING: Running pip as the 'root' user ... lines are red herrings from GitHub Actions log formatting, not the job-failure cause. The real failures are the exit code 2 jobs listed above.

Why

  • The current logic in cuda_pathfinder/cuda/pathfinder/_compatibility_guard_rails.py only accepts CTK metadata from an enclosing version.json or from wheel ownership / cuda-toolkit metadata.
  • The local CI toolkit layout under cuda_toolkit/ appears to provide neither. I also checked the current tree and there is no version.json under cuda_toolkit/.
  • A filename-based guess would not be a proper fix. The local 13.0.2 and 13.2.1 jobs use the same artifact names (libnvrtc.so.13 on Linux, nvrtc64_130_0.dll / nvJitLink_130_0.dll on Windows), so the minor CTK line is not recoverable from the library filename alone.

Proper fix

  • The clean fix is to add a third authoritative CTK metadata source for local-toolkit jobs.
  • Best option: make the local CI cuda_toolkit/ tree include version.json at its root, so the existing pathfinder logic works unchanged.
  • Second-best option: teach pathfinder to read another authoritative manifest already present in that local toolkit tree, if CI already stages one.
  • Least attractive option: add an explicit env-var metadata source for local-toolkit test jobs. That would work, but it is more CI-specific and less clean than shipping real toolkit metadata.

rwgk added 3 commits April 25, 2026 12:12
Introduce a small toolkit-info utility that reads the CUDA_VERSION macro from cuda.h so follow-up guard-rails changes can infer CTK major.minor from toolkit headers without depending on version.json.

Made-with: Cursor
Centralize encoded CUDA version parsing and validation so toolkit and driver version helpers stay aligned and cuda.h parsing gets consistent string conversion and error reporting.

Made-with: Cursor
Replace version.json-based CTK root metadata with cuda.h parsing so compatibility checks use a simpler, more universal toolkit source while preserving wheel-based metadata inference.

Made-with: Cursor
@rwgk
Copy link
Copy Markdown
Contributor Author

rwgk commented Apr 25, 2026

/ok to test

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cuda.pathfinder Everything related to the cuda.pathfinder module feature New feature or request P0 High priority - Must do!

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEA]: Explore pathfinder "scoped search" feature

1 participant