Bug Description
On sm_120 systems with Warp built against CUDA Toolkit 13.1 through 13.4, compiling one CUDA module can cause an unrelated module compiled later in the same process to fail with CUDA error 700.
Both modules compile successfully. The earlier module also runs successfully. The failure occurs when the later kernel is launched:
CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Compiling the later kernel in a fresh process produces the expected result.
The problem occurs after either of these sequences:
- A module compiled at Warp optimization level 0, followed by a module compiled at the default optimization level.
- On Linux, a module compiled with device debugging, followed by a release module.
This makes failures depend on compilation order. A kernel may work by itself but fail when an application or test worker reuses a process that previously compiled a module with different settings. Once the illegal memory access occurs, later CUDA operations in that process may report secondary errors because the CUDA context is already in an error state.
Reproduction
The reduced target accepts a 136-byte structure by value and reads a pointer from it. After a level 0 warm-up compilation, launching the target fails with CUDA error 700. Without the warm-up, it returns the expected value.
Reducing the structure from 136 bytes to 128 bytes makes the failure disappear.
The device-debug case follows the same pattern: compile a debug module, then compile and launch the release target in the same process. Running the release target in a new process passes.
Results by toolkit
Each optimization-level result was repeated in three fresh processes with the compiler and driver caches disabled.
| CUDA Toolkit |
NVRTC |
Target by itself |
Level 0 module, then target |
| 13.0.3 |
13.0.88 |
Pass |
Pass |
| 13.1.2 |
13.1.115 |
Pass |
CUDA 700 |
| 13.2.2 |
13.2.86 |
Pass |
CUDA 700 |
| 13.3.1 |
13.3.33 |
Pass |
CUDA 700 |
| 13.4.1 |
13.4.59 |
Pass |
CUDA 700 |
The regression begins between the tested CUDA 13.0 and 13.1 releases. No fixed CUDA release has been identified yet.
The runtime failure has been confirmed on sm_120. CUDA 13.4.1 CI runs on H100 and L4 GPUs passed.
Generated code
The target source, compile options, and launch arguments are the same in the passing and failing cases. The earlier compilation changes the generated code for the target.
The passing PTX loads from the kernel parameter directly. The failing PTX converts the parameter address and then uses the converted address for indirect parameter-space loads.
The SASS differs in the same area. The passing binary uses fixed constant-parameter offsets, while the failing binary uses register-indexed constant-parameter loads. Correcting the affected PTX parameter loads makes the kernel pass.
A reduced CUDA C++ case reproduces the same compilation-order dependency without Warp.
Temporary Warp workaround
PR #1933 changes Warp optimization level 0 to use the level 1 fast-compilation setting on CUDA 13.1 and newer. Warp emits a warning when this fallback applies.
Full device-debug information remains available. Debug and release CUDA modules must be compiled in separate processes on affected Linux configurations.
The fallback and warning should remain until a CUDA release passes both compilation sequences. This issue can track that verification and the eventual removal of the workaround.
System Information
- Warp 1.18.0 development branch before the workaround
- Python 3.12.13
- Ubuntu 22.04.5
- NVIDIA RTX PRO 6000 Blackwell Server Edition,
sm_120, MIG 1g.24gb
- NVIDIA driver 595.58.03
- CUDA driver API 13.2
Bug Description
On
sm_120systems with Warp built against CUDA Toolkit 13.1 through 13.4, compiling one CUDA module can cause an unrelated module compiled later in the same process to fail with CUDA error 700.Both modules compile successfully. The earlier module also runs successfully. The failure occurs when the later kernel is launched:
Compiling the later kernel in a fresh process produces the expected result.
The problem occurs after either of these sequences:
This makes failures depend on compilation order. A kernel may work by itself but fail when an application or test worker reuses a process that previously compiled a module with different settings. Once the illegal memory access occurs, later CUDA operations in that process may report secondary errors because the CUDA context is already in an error state.
Reproduction
The reduced target accepts a 136-byte structure by value and reads a pointer from it. After a level 0 warm-up compilation, launching the target fails with CUDA error 700. Without the warm-up, it returns the expected value.
Reducing the structure from 136 bytes to 128 bytes makes the failure disappear.
The device-debug case follows the same pattern: compile a debug module, then compile and launch the release target in the same process. Running the release target in a new process passes.
Results by toolkit
Each optimization-level result was repeated in three fresh processes with the compiler and driver caches disabled.
The regression begins between the tested CUDA 13.0 and 13.1 releases. No fixed CUDA release has been identified yet.
The runtime failure has been confirmed on
sm_120. CUDA 13.4.1 CI runs on H100 and L4 GPUs passed.Generated code
The target source, compile options, and launch arguments are the same in the passing and failing cases. The earlier compilation changes the generated code for the target.
The passing PTX loads from the kernel parameter directly. The failing PTX converts the parameter address and then uses the converted address for indirect parameter-space loads.
The SASS differs in the same area. The passing binary uses fixed constant-parameter offsets, while the failing binary uses register-indexed constant-parameter loads. Correcting the affected PTX parameter loads makes the kernel pass.
A reduced CUDA C++ case reproduces the same compilation-order dependency without Warp.
Temporary Warp workaround
PR #1933 changes Warp optimization level 0 to use the level 1 fast-compilation setting on CUDA 13.1 and newer. Warp emits a warning when this fallback applies.
Full device-debug information remains available. Debug and release CUDA modules must be compiled in separate processes on affected Linux configurations.
The fallback and warning should remain until a CUDA release passes both compilation sequences. This issue can track that verification and the eventual removal of the workaround.
System Information
sm_120, MIG 1g.24gb