Installation on Blackwell GPU: Unknown CUDA Architecture Name 12.0 in CUDA_SELECT_NVCC_ARCH_FLAGS

[mxbi]

We're trying to compile k2 from source on a machine with a 5090 and 4090 in it (using CUDA 12.8 / nightly pytorch), but we get within the build output (when doing setup.py install):

build command is:

                cd build/temp.linux-x86_64-cpython-312

                cmake -DCMAKE_BUILD_TYPE=Release -DPYTHON_EXECUTABLE=/research/icefall/.venv/bin/python3 -DK2_ENABLE_BENCHMARK=OFF  -DK2_ENABLE_TESTS=OFF  -DCMAKE_INSTALL_PREFIX=/research/icefall/k2/build/lib.linux-x86_64-cpython-312/k2  -DCMAKE_CXX_STANDARD=17  /research/icefall/k2

                cat k2/csrc/version.h

                make -j6 install

-- CMAKE_VERSION: 3.28.3
CMake Deprecation Warning at CMakeLists.txt:20 (cmake_policy):
  The OLD behavior for policy CMP0111 will be removed from a future version
  of CMake.

  The cmake-policies(7) manual explains that the OLD behaviors of all
  policies are deprecated and that a policy should be set to OLD only under
  specific short-term circumstances.  Projects should be ported to the NEW
  behavior and not rely on setting a policy to OLD.


-- Enabled languages: CXX;CUDA
-- K2_OS: Ubuntu 24.04.2 LTS
-- C++ Standard version: 17
-- Autodetected CUDA architecture(s):  8.9 12.0
CMake Error at cmake/select_compute_arch.cmake:250 (message):
  Unknown CUDA Architecture Name 12.0 in CUDA_SELECT_NVCC_ARCH_FLAGS
Call Stack (most recent call first):
  CMakeLists.txt:226 (cuda_select_nvcc_arch_flags)


CMake Error at cmake/select_compute_arch.cmake:254 (message):
  arch_bin wasn't set for some reason
Call Stack (most recent call first):
  CMakeLists.txt:226 (cuda_select_nvcc_arch_flags)


-- K2_COMPUTE_ARCH_FLAGS: -gencode;arch=compute_89,code=sm_89
-- K2_COMPUTE_ARCH_CANDIDATES 50;60;61;70;75;80;86;90
-- Skipping arch 50
-- Skipping arch 60
-- Skipping arch 61
-- Skipping arch 70
-- Skipping arch 75
-- Skipping arch 80
-- Skipping arch 86
-- Skipping arch 90
-- K2_COMPUTE_ARCHS: 50;60;61;70;75;80;86;90

Looks like select_compute_arch.cmake doesn't support CUDA architecture 12.0 which the blackwell cards use. Is there an obvious way to get this working on Blackwell GPUs, or is there anything we can do to help get this support working?

Thanks!

[csukuangfj]

Please use
#1333

or you can use our pre-built k2 wheels for cuda 12.8 + nightly torch
see https://k2-fsa.github.io/k2/cuda.html

[mxbi]

Thanks for such a quick reply!

Using the PR branch, I still get:

-- Autodetected CUDA architecture(s):  8.9 12.0
CMake Error at cmake/select_compute_arch.cmake:250 (message):
  Unknown CUDA Architecture Name 12.0 in CUDA_SELECT_NVCC_ARCH_FLAGS
Call Stack (most recent call first):
  CMakeLists.txt:226 (cuda_select_nvcc_arch_flags)


CMake Error at cmake/select_compute_arch.cmake:254 (message):
  arch_bin wasn't set for some reason
Call Stack (most recent call first):
  CMakeLists.txt:226 (cuda_select_nvcc_arch_flags)

on compile. Looks like an issue with the compute capability being >9 means that this regex fails:
https://github.com/k2-fsa/k2/blob/master/cmake/select_compute_arch.cmake#L222

I tried changing it to match [0-9]+ instead of just [0-9], following upstream, which gets further but gives me a second error.

-- PyTorch cuda version: 12.8
CMake Error at cmake/torch.cmake:88 (set_property):
  set_property could not find TARGET torch_cuda.  Perhaps it has not yet been
  created.
Call Stack (most recent call first):
  build/temp.linux-x86_64-cpython-312/_deps/kaldifeat-src/CMakeLists.txt:62 (include)


CMake Error at cmake/torch.cmake:92 (set_property):
  set_property could not find TARGET torch_cpu.  Perhaps it has not yet been
  created.
Call Stack (most recent call first):
  build/temp.linux-x86_64-cpython-312/_deps/kaldifeat-src/CMakeLists.txt:62 (include)

I'm going to try the pre-built wheels now (thank you!), but I think there might still be an issue here with building on compute capability 12.0

[csukuangfj]

please see

k2/scripts/github_actions/build-ubuntu-cuda.sh

Lines 26 to 27 in 0a09f67

	# see https://github.com/pytorch/pytorch/issues/113948
	export TORCH_CUDA_ARCH_LIST="8.0 8.6 8.9 9.0"

[mxbi]

I was able to get it working with the pre-built nightly binaries, both on Blackwell and Ada, thank you! Tested egs/yesno/ASR/tdnn/train.py.

For anyone else, the full set of commands I used from a fresh venv (using uv):

# Torchaudio has to be installed first, it'll install an incorrect version of torch which then gets overwritten in the next command
# There are no matching pairs of packages according to the uv resolver so we need this hack
uv pip install torchaudio --index-url  https://download.pytorch.org/whl/nightly/cu128
uv pip install torch=='2.7.0.dev20250303+cu128' --index-url https://download.pytorch.org/whl/nightly/cu128
uv pip install https://huggingface.co/csukuangfj/k2/resolve/main/ubuntu-cuda/k2-1.24.4.dev20250304+cuda12.8.torch2.7.0.dev20250303-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
pip install git+https://github.com/lhotse-speech/lhotse

git clone https://github.com/k2-fsa/icefall
cd icefall
uv pip install -e .

If I need to do any source changes, i'll investigate compiling on Blackwell further!