Current behavior
FlashInfer pip wheels are currently agnostic to the installed CUDA version, which can lead to compatibility issues.
FlashInfer depends on precompiled kernels generated from TRT-LLM and cuDNN, pulling cubins produced for the CUDA 12 toolkit via TRT-LLM, while cuDNN is sourced from nvidia-cudnn-cu13. In environments with CUDA 12 installed, this can cause cuDNN incompatibilities, and in environments with CUDA 13 installed, the TRT-LLM kernels may fail.
Requirement for cuda major version
FlashInfer must explicitly support both CUDA 12 and CUDA 13 toolkits, ensuring that only cubins with binary compatibility for the corresponding CUDA major version are loaded and executed. This alignment is necessary to prevent runtime failures and to guarantee stable behavior across supported CUDA configurations.
Guidance
A potential solution is to have flashinfer-python always target the latest supported CUDA major toolkit, while introducing a flashinfer-python-cu<current_latest_minus_1> variant that depends on the appropriate cubins/wheels for the previous major version. This approach mirrors common CUDA-aware packaging patterns and provides clear, version-specific wheels for users to select based on their environment
Current behavior
FlashInfer pip wheels are currently agnostic to the installed CUDA version, which can lead to compatibility issues.
FlashInfer depends on precompiled kernels generated from TRT-LLM and cuDNN, pulling cubins produced for the CUDA 12 toolkit via TRT-LLM, while cuDNN is sourced from
nvidia-cudnn-cu13. In environments with CUDA 12 installed, this can cause cuDNN incompatibilities, and in environments with CUDA 13 installed, the TRT-LLM kernels may fail.Requirement for cuda major version
FlashInfer must explicitly support both CUDA 12 and CUDA 13 toolkits, ensuring that only cubins with binary compatibility for the corresponding CUDA major version are loaded and executed. This alignment is necessary to prevent runtime failures and to guarantee stable behavior across supported CUDA configurations.
Guidance
A potential solution is to have
flashinfer-pythonalways target the latest supported CUDA major toolkit, while introducing aflashinfer-python-cu<current_latest_minus_1>variant that depends on the appropriate cubins/wheels for the previous major version. This approach mirrors common CUDA-aware packaging patterns and provides clear, version-specific wheels for users to select based on their environment