Skip to content

tests: skip local NVML runtime mismatches while preserving CI failures#1739

Open
cpcloud wants to merge 3 commits intoNVIDIA:mainfrom
cpcloud:fix/nvml-local-skip-ci-fail
Open

tests: skip local NVML runtime mismatches while preserving CI failures#1739
cpcloud wants to merge 3 commits intoNVIDIA:mainfrom
cpcloud:fix/nvml-local-skip-ci-fail

Conversation

@cpcloud
Copy link
Contributor

@cpcloud cpcloud commented Mar 7, 2026

Summary

  • Centralize NVML runtime gating through require_nvml_runtime_or_skip_local fixtures in cuda_bindings and cuda_core tests.
  • Treat NVML init/load failures (including driver/library mismatch and missing NVML shared library) as local skips, but continue to fail in CI by re-raising.
  • Cover the behavior with regression tests and apply fixture-based gating across NVML-dependent test modules/fixtures.
  • Rationale: after a driver upgrade without a reboot (a common local developer state), NVML can report a temporary driver/library mismatch; local runs should skip NVML-dependent tests instead of failing collection, while CI should still fail fast for real infra regressions.

Test plan

  • pixi run --manifest-path cuda_bindings pytest cuda_bindings/tests --override-ini norecursedirs=examples -k "not test_cufile"
  • CI=1 pixi run --manifest-path cuda_bindings pytest cuda_bindings/tests/nvml/test_init.py::test_init_ref_count (expected error on NVML mismatch in CI mode)
  • pixi run --manifest-path cuda_core test (currently blocked in this workspace by unrelated import mismatch: cuda.core._resource_handles does not export expected C function create_culink_handle)
  • CI=1 pixi run --manifest-path cuda_core pytest cuda_core/tests/system/test_system_system.py::test_num_devices (same unrelated import mismatch blocker)

Made with Cursor

@copy-pr-bot
Copy link
Contributor

copy-pr-bot bot commented Mar 7, 2026

Auto-sync is disabled for ready for review pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

cpcloud added 3 commits March 7, 2026 11:51
Driver upgrades without a reboot can temporarily leave NVML in a driver/library mismatch state, which is a common local developer scenario. Route NVML-dependent checks through shared fixtures/helpers so local runs skip cleanly while CI still fails fast on real NVML init/load regressions.

Made-with: Cursor
Run repository hooks and keep the NVML fixture changes compliant by applying ruff import ordering and formatting adjustments.

Made-with: Cursor
Apply hook-driven import ordering/spacing updates introduced by rebasing onto upstream/main so pre-commit passes cleanly.

Made-with: Cursor
@cpcloud cpcloud force-pushed the fix/nvml-local-skip-ci-fail branch from 3ae1ece to 4272269 Compare March 7, 2026 16:52
@cpcloud
Copy link
Contributor Author

cpcloud commented Mar 7, 2026

/ok to test

@github-actions
Copy link

github-actions bot commented Mar 7, 2026

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant