Make checkpoint tests fail on missing required binding symbols

[rwgk]

Closes #2149

Summary

• Tighten the cuda.core checkpoint test availability guard so it still skips true unsupported environments, but no longer skips missing required cuda.bindings symbols.
• Add a focused cuda.bindings completeness test for the checkpoint symbols required by cuda.core.checkpoint, including CUcheckpointRestoreArgs.
• Cover the skip-gate behavior directly so old cuda-bindings versions and unsupported installed drivers remain skippable, while missing required bindings fail.

Context

This is a follow-up to #2144 and fixes the test coverage gap tracked in #2149.

The CUDA 13.3.0 CUcheckpointRestoreArgs generation issue fixed by #2144 could pass the existing test flow because the cuda.core checkpoint tests treated all RuntimeErrors from checkpoint._get_driver() as an unsupported environment. That included:

CUDA checkpointing requires cuda.bindings with CUDA checkpoint API support. Missing: CUcheckpointRestoreArgs

This PR keeps the intended skips for genuinely unsupported configurations, but lets missing required binding attributes propagate as test failures.

Validation

On the pre-#2144 base, these focused tests now expose the breakage:

pytest cuda_core/tests/test_checkpoint.py

fails during collection with:

RuntimeError: CUDA checkpointing requires cuda.bindings with CUDA checkpoint API support. Missing: CUcheckpointRestoreArgs

and:

pytest cuda_bindings/tests/test_cuda.py::test_cuCheckpoint_required_bindings_present

fails with:

missing == ['CUcheckpointRestoreArgs']

After PR #2144 lands and this branch is rebased onto it, the focused checkpoint tests should pass and demonstrate that the original generation issue is fixed while the error-masking skip is closed.

Related

• Follow-up to Fix CUDA checkpoint restore args and generated API docs #2144.
• Related backport: Backport checkpoint restore arg layout handling to 12.9.x #2145.
• Related generator-side follow-up: cybind MR 396.
• See also: nvbug 6097301 comment 34.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[rwgk]

/ok to test

[github-actions]

Doc Preview CI
🚀 View preview at https://nvidia.github.io/cuda-python/pr-preview/pr-2150/
https://nvidia.github.io/cuda-python/pr-preview/pr-2150/cuda-core/
https://nvidia.github.io/cuda-python/pr-preview/pr-2150/cuda-bindings/
https://nvidia.github.io/cuda-python/pr-preview/pr-2150/cuda-pathfinder/
Preview will be ready when the GitHub Pages deployment is complete.

[rwgk]

PR 2150 first CI failure analysis

Workflow: https://github.com/NVIDIA/cuda-python/actions/runs/26591678170

Commit: 293258d

Workflow result: failed.

High-level result

The build and non-test infrastructure mostly passed:

• Build jobs passed.
• Docs passed.
• pre-commit.ci passed.
• The final Check job status job failed because matrix test jobs failed.

The failures are concentrated in test matrix jobs. There were 37 failed test jobs plus the final status aggregation job.

Failure counts by CUDA version:

• CUDA 13.3.0: 24 failed test jobs.
• CUDA 12.9.1: 13 failed test jobs.
• CUDA 13.0.2: no failures observed in the failed-job list.

Failure counts by platform:

• linux-64: 17 failed test jobs.
• linux-aarch64: 12 failed test jobs.
• win-64: 8 failed test jobs.

Failure mode 1: CUDA 13.3 missing CUcheckpointRestoreArgs

This is the expected proof-of-coverage failure for the issue fixed by PR #2144.

Representative failed jobs:

• https://github.com/NVIDIA/cuda-python/actions/runs/26591678170/job/78353772395
• https://github.com/NVIDIA/cuda-python/actions/runs/26591678170/job/78354692898

Observed failure:

tests/test_cuda.py::test_cuCheckpoint_required_bindings_present FAILED
E       AssertionError: assert ['CUcheckpointRestoreArgs'] == []
E         Left contains one more item: 'CUcheckpointRestoreArgs'

This shows the new cuda_bindings/tests/test_cuda.py::test_cuCheckpoint_required_bindings_present test catches the missing generated binding directly. This is exactly the failure PR #2150 was intended to expose before PR #2144 is merged/rebased in.

Failure mode 2: Linux CUDA 12.9 missing CUcheckpointGpuPair

This is a separate checkpoint binding gap surfaced by tightening the cuda.core checkpoint availability guard.

Representative failed jobs:

• https://github.com/NVIDIA/cuda-python/actions/runs/26591678170/job/78353772318
• https://github.com/NVIDIA/cuda-python/actions/runs/26591678170/job/78354682523

Observed failure:

ERROR collecting tests/test_checkpoint.py
RuntimeError: CUDA checkpointing requires cuda.bindings with CUDA checkpoint API support. Missing: CUcheckpointGpuPair
binding_ver = (12, 9, 7)
missing    = ['CUcheckpointGpuPair']

This happens during cuda_core/tests/test_checkpoint.py collection on Linux. Windows does not hit this path because the checkpoint tests are platform-skipped before _checkpoint_available() is evaluated.

This is not the CUDA 13.3 CUcheckpointRestoreArgs issue. It means the tightened guard also exposes that CUDA 12.9 Linux bindings report enough checkpoint API surface to reach cuda.core.checkpoint._get_driver(), but still lack CUcheckpointGpuPair, which cuda.core considers required.

Interpretation

CI behaved as intended for the main goal: PR #2150 converts the previous skip-masked CUDA 13.3 binding regression into clear test failures.

The run also identifies a follow-up decision for CUDA 12.9:

• If CUDA 12.9 should support the cuda.core.checkpoint mapping helpers, then CUcheckpointGpuPair needs to be present in the CUDA 12.9 bindings.
• If CUDA 12.9 should not support that surface, the cuda.core availability check needs to classify this specific older-binding condition as skippable rather than as a binding regression.

After PR #2144 is merged and PR #2150 is rebased onto it, the CUDA 13.3 CUcheckpointRestoreArgs failures should disappear. The CUDA 12.9 CUcheckpointGpuPair failures may remain unless they are handled separately.

[rwgk]

I looked into the CUDA 12.9 failures from the first PR #2150 CI run.

The short version: these failures look separate from the CUDA 13.3 CUcheckpointRestoreArgs regression that PR #2144 fixes.

In /usr/local/cuda-12.9, I do not see CUcheckpointGpuPair at all. Even

grep -r -i GpuPair /usr/local/cuda-12.9

returns no matches.

The CUDA 12.9 cuda.h checkpoint restore args are still the older reserved-only layout:

typedef struct CUcheckpointRestoreArgs_st {
    cuuint64_t reserved[8]; /**< Reserved for future use, must be zeroed */
} CUcheckpointRestoreArgs;

That matches the CUDA 12.9 CI failure mode from https://github.com/NVIDIA/cuda-python/actions/runs/26591678170: Linux CUDA 12.9 jobs now fail during cuda_core/tests/test_checkpoint.py collection with:

RuntimeError: CUDA checkpointing requires cuda.bindings with CUDA checkpoint API support. Missing: CUcheckpointGpuPair
binding_ver = (12, 9, 7)
missing    = ['CUcheckpointGpuPair']

So my current interpretation is:

• The CUDA 13.3 failures are expected and useful: PR Make checkpoint tests fail on missing required binding symbols #2150 proves that missing CUcheckpointRestoreArgs would now fail loudly instead of being skip-masked.
• The CUDA 12.9 failures are a separate compatibility issue surfaced by the tighter guard.
• Since CUcheckpointGpuPair does not appear to exist in the CUDA 12.9 headers, this is not a missing 12.9 Python binding. It is more likely that cuda.core.checkpoint is treating CUcheckpointGpuPair as required too broadly for CUDA 12.9.

Possible follow-up direction: keep missing required symbols as failures for APIs that should exist in the active CUDA version, but treat the CUDA 12.9/no-CUcheckpointGpuPair path as an older checkpoint API shape that should remain skippable or should avoid enabling the GPU remapping surface.

[rwgk]

/ok to test

[rwgk]

PR 2150 second CI failure analysis

Workflow: https://github.com/NVIDIA/cuda-python/actions/runs/26596635176

Commit: cd730c1

Current workflow state at inspection time:

• 65 jobs passed.
• 23 jobs failed.
• 1 job was still queued: Test win-64 / Python 3.14, CUDA 13.3.0 (wheels), GPU h100 (x2) (MCDM).

High-level result

The second CI run matches expectations after splitting baseline checkpoint support from GPU-remapping support.

All completed failures are CUDA 13.3.0 test jobs. CUDA 12.9.1 and CUDA 13.0.2 jobs that completed are passing.

Failure counts by CUDA version:

• CUDA 13.3.0: 23 failed test jobs.
• CUDA 12.9.1: 0 failed test jobs observed.
• CUDA 13.0.2: 0 failed test jobs observed.

Failure counts by platform:

• linux-64: 10 failed test jobs, all CUDA 13.3.0.
• linux-aarch64: 6 failed test jobs, all CUDA 13.3.0.
• win-64: 7 failed test jobs, all CUDA 13.3.0.

Remaining failure mode: CUDA 13.3 missing CUcheckpointRestoreArgs

Representative failed jobs:

• https://github.com/NVIDIA/cuda-python/actions/runs/26596635176/job/78371061954
• https://github.com/NVIDIA/cuda-python/actions/runs/26596635176/job/78372022514

Both sampled logs show the same intended failure:

tests/test_cuda.py::test_cuCheckpoint_required_bindings_present FAILED
E       AssertionError: assert ['CUcheckpointRestoreArgs'] == []
E         Left contains one more item: 'CUcheckpointRestoreArgs'
missing    = ['CUcheckpointRestoreArgs']

This is the expected pre-PR-2144 signal. PR #2150 is still proving that the CUDA 13.3 CUcheckpointRestoreArgs binding regression would fail loudly instead of being skip-masked.

Fixed compared with the first CI run

The CUDA 12.9 CUcheckpointGpuPair failure mode is gone.

In the first run, Linux CUDA 12.9 jobs failed during cuda_core/tests/test_checkpoint.py collection with:

RuntimeError: CUDA checkpointing requires cuda.bindings with CUDA checkpoint API support. Missing: CUcheckpointGpuPair
binding_ver = (12, 9, 7)
missing    = ['CUcheckpointGpuPair']

After commit cd730c1, completed CUDA 12.9.1 jobs are passing. This confirms that baseline checkpoint tests no longer require the CUDA 13.x GPU-remapping type, while the CUDA 13.3 missing-CUcheckpointRestoreArgs regression remains a hard failure.

Interpretation

The results are the desired intermediate state:

• PR 2150 no longer over-requires CUcheckpointGpuPair for CUDA 12.9.
• PR 2150 still catches the original CUDA 13.3 CUcheckpointRestoreArgs generation issue.
• After PR 2144 lands and PR 2150 is rebased onto it, the CUDA 13.3 failures should disappear.