Unknown CUDA arch 10.1 when building C++ extension against 2.9.0+cu128

[SuibianP]

🐛 Describe the bug

Explicitly specifying 10.1 in TORCH_CUDA_ARCH_LIST makes extension build on torch 2.9.0 and CUDA < 13.0 (tested on 12.8 and 12.9) fail with ValueError: Unknown CUDA arch (10.1) or GPU not supported from utils/cpp_extension.py:_get_cuda_arch_flags. This is presumably related to the renaming of sm_101 to sm_110 in CUDA 13.0 (#156176, 9c5601e).

FROM pytorch/pytorch:2.9.0-cuda12.8-cudnn9-devel
RUN apt-get update && apt-get install -y curl
RUN curl -sSL https://github.com/pytorch/extension-cpp/archive/0ec4969.tar.gz | tar -xvz && \
        sed -i 's/torch.cuda.is_available()/True/g' extension-cpp-0ec4969c7bc8e15a8456e5eb9d9ca0a7ec15bc95/setup.py
WORKDIR extension-cpp-0ec4969c7bc8e15a8456e5eb9d9ca0a7ec15bc95
ENV TORCH_CUDA_ARCH_LIST=10.1
RUN pip install --break-system-packages --no-build-isolation -e .

Building wheels for collected packages: extension_cpp
  Building wheel for extension_cpp (pyproject.toml): started
  Building wheel for extension_cpp (pyproject.toml): finished with status 'error'
  error: subprocess-exited-with-error

  × Building wheel for extension_cpp (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [85 lines of output]
      W1023 08:18:50.032000 91 site-packages/torch/utils/cpp_extension.py:117] No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
      running bdist_wheel
      running build
      running build_py
      creating build/lib.linux-x86_64-cpython-311/extension_cpp
      copying extension_cpp/__init__.py -> build/lib.linux-x86_64-cpython-311/extension_cpp
      copying extension_cpp/ops.py -> build/lib.linux-x86_64-cpython-311/extension_cpp
      running build_ext
      W1023 08:18:50.078000 91 site-packages/torch/utils/cpp_extension.py:531] There are no g++ version bounds defined for CUDA version 12.8
      building 'extension_cpp._C' extension
      creating /workspace/extension-cpp-0ec4969c7bc8e15a8456e5eb9d9ca0a7ec15bc95/build/temp.linux-x86_64-cpython-311/extension_cpp/csrc/cuda
      Traceback (most recent call last):
        File "/opt/conda/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 389, in <module>
          main()
        File "/opt/conda/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 373, in main
          json_out["return_val"] = hook(**hook_input["kwargs"])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/conda/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 280, in build_wheel
          return _build_backend().build_wheel(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/conda/lib/python3.11/site-packages/setuptools/build_meta.py", line 435, in build_wheel
          return _build(['bdist_wheel', '--dist-info-dir', str(metadata_directory)])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/conda/lib/python3.11/site-packages/setuptools/build_meta.py", line 423, in _build
          return self._build_with_temp_dir(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/conda/lib/python3.11/site-packages/setuptools/build_meta.py", line 404, in _build_with_temp_dir
          self.run_setup()
        File "/opt/conda/lib/python3.11/site-packages/setuptools/build_meta.py", line 317, in run_setup
          exec(code, locals())
        File "<string>", line 75, in <module>
        File "/opt/conda/lib/python3.11/site-packages/setuptools/__init__.py", line 117, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/conda/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 186, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "/opt/conda/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 202, in run_commands
          dist.run_commands()
        File "/opt/conda/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 1002, in run_commands
          self.run_command(cmd)
        File "/opt/conda/lib/python3.11/site-packages/setuptools/dist.py", line 1104, in run_command
          super().run_command(command)
        File "/opt/conda/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
          cmd_obj.run()
        File "/opt/conda/lib/python3.11/site-packages/setuptools/command/bdist_wheel.py", line 370, in run
          self.run_command("build")
        File "/opt/conda/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 357, in run_command
          self.distribution.run_command(command)
        File "/opt/conda/lib/python3.11/site-packages/setuptools/dist.py", line 1104, in run_command
          super().run_command(command)
        File "/opt/conda/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
          cmd_obj.run()
        File "/opt/conda/lib/python3.11/site-packages/setuptools/_distutils/command/build.py", line 135, in run
          self.run_command(cmd_name)
        File "/opt/conda/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 357, in run_command
          self.distribution.run_command(command)
        File "/opt/conda/lib/python3.11/site-packages/setuptools/dist.py", line 1104, in run_command
          super().run_command(command)
        File "/opt/conda/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
          cmd_obj.run()
        File "/opt/conda/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 99, in run
          _build_ext.run(self)
        File "/opt/conda/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 368, in run
          self.build_extensions()
        File "/opt/conda/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1082, in build_extensions
          build_ext.build_extensions(self)
        File "/opt/conda/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 484, in build_extensions
          self._build_extensions_serial()
        File "/opt/conda/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 510, in _build_extensions_serial
          self.build_extension(ext)
        File "/opt/conda/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1082, in build_extensions
          build_ext.build_extensions(self)
        File "/opt/conda/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 484, in build_extensions
          self._build_extensions_serial()
        File "/opt/conda/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 510, in _build_extensions_serial
          self.build_extension(ext)
        File "/opt/conda/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 264, in build_extension
          _build_ext.build_extension(self, ext)
        File "/opt/conda/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 565, in build_extension
          objects = self.compiler.compile(
                    ^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/conda/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 825, in unix_wrap_ninja_compile
          cuda_post_cflags = unix_cuda_flags(cuda_post_cflags)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/conda/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 723, in unix_cuda_flags
          cflags + _get_cuda_arch_flags(cflags))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/conda/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 2460, in _get_cuda_arch_flags
          raise ValueError(f"Unknown CUDA arch ({arch}) or GPU not supported")
      ValueError: Unknown CUDA arch (10.1) or GPU not supported
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for extension_cpp
Failed to build extension_cpp
ERROR: Failed to build installable wheels for some pyproject.toml based projects (extension_cpp)

Versions

Collecting environment information...
PyTorch version: 2.9.0+cu128
Is debug build: False
CUDA used to build PyTorch: 12.8
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.5 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 4.1.0
Libc version: glibc-2.35

Python version: 3.11.14 | packaged by conda-forge | (main, Oct 13 2025, 14:09:32) [GCC 14.3.0] (64-bit runtime)
Python platform: Linux-6.11.0-1013-nvidia-x86_64-with-glibc2.35
Is CUDA available: False
CUDA runtime version: 12.8.93
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Could not collect
Is XPU available: False
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 43 bits physical, 57 bits virtual
Byte Order: Little Endian
CPU(s): 80
On-line CPU(s) list: 0-79
Vendor ID: GenuineIntel
Model name: Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz
CPU family: 6
Model: 106
Thread(s) per core: 2
Core(s) per socket: 20
Socket(s): 2
Stepping: 6
BogoMIPS: 3990.62 08:08:14 [62/1932]
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx pdpe1gb
rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq dtes64 ssse3 fma cx16 pdcm pcid sse4_1 sse4_2 x2apic mov
be popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep
bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves wbnoinvd arat avx512vbmi umip
pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid fsrm md_clear arch_capabilities
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 2.5 MiB (80 instances)
L1i cache: 2.5 MiB (80 instances)
L2 cache: 160 MiB (40 instances)
L3 cache: 32 MiB (2 instances)
NUMA node(s): 2
NUMA node0 CPU(s): 0-39
NUMA node1 CPU(s): 40-79
Vulnerability Gather data sampling: Unknown: Dependent on hypervisor status
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS SW sequence; BHI SW loop, KVM SW loop
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Mitigation; TSX disabled

Versions of relevant libraries:
[pip3] numpy==2.3.3
[pip3] nvidia-cublas-cu12==12.8.4.1
[pip3] nvidia-cuda-cupti-cu12==12.8.90
[pip3] nvidia-cuda-nvrtc-cu12==12.8.93
[pip3] nvidia-cuda-runtime-cu12==12.8.90
[pip3] nvidia-cudnn-cu12==9.10.2.21
[pip3] nvidia-cufft-cu12==11.3.3.83
[pip3] nvidia-curand-cu12==10.3.9.90
[pip3] nvidia-cusolver-cu12==11.7.3.90
[pip3] nvidia-cusparse-cu12==12.5.8.93
[pip3] nvidia-cusparselt-cu12==0.7.1
[pip3] nvidia-nccl-cu12==2.27.5
[pip3] nvidia-nvjitlink-cu12==12.8.93
[pip3] nvidia-nvtx-cu12==12.8.90
[pip3] optree==0.17.0
[pip3] torch==2.9.0+cu128
[pip3] torchaudio==2.9.0+cu128
[pip3] torchelastic==0.2.2
[pip3] torchvision==0.24.0+cu128
[pip3] triton==3.5.0
[conda] numpy 2.3.3 py311h2e04523_0 conda-forge
[conda] nvidia-cublas-cu12 12.8.4.1 pypi_0 pypi
[conda] nvidia-cuda-cupti-cu12 12.8.90 pypi_0 pypi
[conda] nvidia-cuda-nvrtc-cu12 12.8.93 pypi_0 pypi
[conda] nvidia-cuda-runtime-cu12 12.8.90 pypi_0 pypi
[conda] nvidia-cudnn-cu12 9.10.2.21 pypi_0 pypi
[conda] nvidia-cufft-cu12 11.3.3.83 pypi_0 pypi
[conda] nvidia-curand-cu12 10.3.9.90 pypi_0 pypi
[conda] nvidia-cusolver-cu12 11.7.3.90 pypi_0 pypi
[conda] nvidia-cusparse-cu12 12.5.8.93 pypi_0 pypi
[conda] nvidia-cusparselt-cu12 0.7.1 pypi_0 pypi
[conda] nvidia-nccl-cu12 2.27.5 pypi_0 pypi
[conda] nvidia-nvjitlink-cu12 12.8.93 pypi_0 pypi
[conda] nvidia-nvtx-cu12 12.8.90 pypi_0 pypi
[conda] optree 0.17.0 pypi_0 pypi
[conda] torch 2.9.0+cu128 pypi_0 pypi
[conda] torchaudio 2.9.0+cu128 pypi_0 pypi
[conda] torchelastic 0.2.2 pypi_0 pypi
[conda] torchvision 0.24.0+cu128 pypi_0 pypi
[conda] triton 3.5.0 pypi_0 pypi

cc @janeyx99 @ptrblck @msaroufim @eqy @jerryzh168 @tinglvv @nWEIdia @malfet @zou3519

[eqy]

CC @nWEIdia @tinglvv

[nWEIdia]

What happens if you change to ENV TORCH_CUDA_ARCH_LIST=11.0?

[SuibianP]

@nWEIdia This fails differently - nvcc fatal : Unsupported gpu architecture 'compute_110' instead.

Logs

Building wheels for collected packages: extension_cpp
  Building editable for extension_cpp (pyproject.toml): started
  Building editable for extension_cpp (pyproject.toml): finished with status 'error'
  error: subprocess-exited-with-error
  
  × Building editable for extension_cpp (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [180 lines of output]
      W1118 13:26:20.077000 14 site-packages/torch/utils/cpp_extension.py:117] No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
      running editable_wheel
      creating /tmp/pip-wheel-qbztqh0g/.tmp-k7tdtms4/extension_cpp.egg-info
      writing /tmp/pip-wheel-qbztqh0g/.tmp-k7tdtms4/extension_cpp.egg-info/PKG-INFO
      writing dependency_links to /tmp/pip-wheel-qbztqh0g/.tmp-k7tdtms4/extension_cpp.egg-info/dependency_links.txt
      writing requirements to /tmp/pip-wheel-qbztqh0g/.tmp-k7tdtms4/extension_cpp.egg-info/requires.txt
      writing top-level names to /tmp/pip-wheel-qbztqh0g/.tmp-k7tdtms4/extension_cpp.egg-info/top_level.txt
      writing manifest file '/tmp/pip-wheel-qbztqh0g/.tmp-k7tdtms4/extension_cpp.egg-info/SOURCES.txt'
      reading manifest file '/tmp/pip-wheel-qbztqh0g/.tmp-k7tdtms4/extension_cpp.egg-info/SOURCES.txt'
      writing manifest file '/tmp/pip-wheel-qbztqh0g/.tmp-k7tdtms4/extension_cpp.egg-info/SOURCES.txt'
      creating '/tmp/pip-wheel-qbztqh0g/.tmp-k7tdtms4/extension_cpp-0.0.1.dist-info'
      creating /tmp/pip-wheel-qbztqh0g/.tmp-k7tdtms4/extension_cpp-0.0.1.dist-info/WHEEL
      running build_py
      running build_ext
      W1118 13:26:20.183000 14 site-packages/torch/utils/cpp_extension.py:531] There are no g++ version bounds defined for CUDA version 12.8
      building 'extension_cpp._C' extension
      creating /tmp/tmpc714rgsu.build-temp/extension_cpp/csrc/cuda
      [1/2] /usr/local/cuda/bin/nvcc --generate-dependencies-with-compile --dependency-output /tmp/tmpc714rgsu.build-temp/extension_cpp/csrc/cuda/muladd.o.d -I/opt/conda/lib/python3.11/site-packages/torch/include -I/opt/conda/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/usr/local/cuda/include -I/opt/conda/include/python3.11 -c -c /workspace/extension-cpp-0ec4969c7bc8e15a8456e5eb9d9ca0a7ec15bc95/extension_cpp/csrc/cuda/muladd.cu -o /tmp/tmpc714rgsu.build-temp/extension_cpp/csrc/cuda/muladd.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -O3 -DTORCH_API_INCLUDE_EXTENSION_H -DPy_LIMITED_API=0x03090000 -DTORCH_EXTENSION_NAME=_C -gencode=arch=compute_110,code=sm_110 -std=c++17
      FAILED: [code=1] /tmp/tmpc714rgsu.build-temp/extension_cpp/csrc/cuda/muladd.o
      /usr/local/cuda/bin/nvcc --generate-dependencies-with-compile --dependency-output /tmp/tmpc714rgsu.build-temp/extension_cpp/csrc/cuda/muladd.o.d -I/opt/conda/lib/python3.11/site-packages/torch/include -I/opt/conda/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/usr/local/cuda/include -I/opt/conda/include/python3.11 -c -c /workspace/extension-cpp-0ec4969c7bc8e15a8456e5eb9d9ca0a7ec15bc95/extension_cpp/csrc/cuda/muladd.cu -o /tmp/tmpc714rgsu.build-temp/extension_cpp/csrc/cuda/muladd.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -O3 -DTORCH_API_INCLUDE_EXTENSION_H -DPy_LIMITED_API=0x03090000 -DTORCH_EXTENSION_NAME=_C -gencode=arch=compute_110,code=sm_110 -std=c++17
      nvcc fatal   : Unsupported gpu architecture 'compute_110'
      [2/2] c++ -MMD -MF /tmp/tmpc714rgsu.build-temp/extension_cpp/csrc/muladd.o.d -pthread -B /opt/conda/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /opt/conda/include -fPIC -O2 -isystem /opt/conda/include -fPIC -I/opt/conda/lib/python3.11/site-packages/torch/include -I/opt/conda/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/usr/local/cuda/include -I/opt/conda/include/python3.11 -c -c /workspace/extension-cpp-0ec4969c7bc8e15a8456e5eb9d9ca0a7ec15bc95/extension_cpp/csrc/muladd.cpp -o /tmp/tmpc714rgsu.build-temp/extension_cpp/csrc/muladd.o -O3 -fdiagnostics-color=always -DPy_LIMITED_API=0x03090000 -DTORCH_API_INCLUDE_EXTENSION_H -DPy_LIMITED_API=0x03090000 -DTORCH_EXTENSION_NAME=_C -std=c++17
      FAILED: [code=1] /tmp/tmpc714rgsu.build-temp/extension_cpp/csrc/muladd.o
      c++ -MMD -MF /tmp/tmpc714rgsu.build-temp/extension_cpp/csrc/muladd.o.d -pthread -B /opt/conda/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /opt/conda/include -fPIC -O2 -isystem /opt/conda/include -fPIC -I/opt/conda/lib/python3.11/site-packages/torch/include -I/opt/conda/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/usr/local/cuda/include -I/opt/conda/include/python3.11 -c -c /workspace/extension-cpp-0ec4969c7bc8e15a8456e5eb9d9ca0a7ec15bc95/extension_cpp/csrc/muladd.cpp -o /tmp/tmpc714rgsu.build-temp/extension_cpp/csrc/muladd.o -O3 -fdiagnostics-color=always -DPy_LIMITED_API=0x03090000 -DTORCH_API_INCLUDE_EXTENSION_H -DPy_LIMITED_API=0x03090000 -DTORCH_EXTENSION_NAME=_C -std=c++17
      c++: fatal error: Killed signal terminated program cc1plus
      compilation terminated.
      ninja: build stopped: subcommand failed.
      Traceback (most recent call last):
        File "/opt/conda/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 2597, in _run_ninja_build
          subprocess.run(
        File "/opt/conda/lib/python3.11/subprocess.py", line 571, in run
          raise CalledProcessError(retcode, process.args,
      subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
      
      The above exception was the direct cause of the following exception:
      
      Traceback (most recent call last):
        File "/opt/conda/lib/python3.11/site-packages/setuptools/command/editable_wheel.py", line 139, in run
          self._create_wheel_file(bdist_wheel)
        File "/opt/conda/lib/python3.11/site-packages/setuptools/command/editable_wheel.py", line 340, in _create_wheel_file
          files, mapping = self._run_build_commands(dist_name, unpacked, lib, tmp)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/conda/lib/python3.11/site-packages/setuptools/command/editable_wheel.py", line 263, in _run_build_commands
          self._run_build_subcommands()
        File "/opt/conda/lib/python3.11/site-packages/setuptools/command/editable_wheel.py", line 290, in _run_build_subcommands
          self.run_command(name)
        File "/opt/conda/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 357, in run_command
          self.distribution.run_command(command)
        File "/opt/conda/lib/python3.11/site-packages/setuptools/dist.py", line 1104, in run_command
          super().run_command(command)
        File "/opt/conda/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
          cmd_obj.run()
        File "/opt/conda/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 99, in run
          _build_ext.run(self)
        File "/opt/conda/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 368, in run
          self.build_extensions()
        File "/opt/conda/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1082, in build_extensions
          build_ext.build_extensions(self)
        File "/opt/conda/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 484, in build_extensions
          self._build_extensions_serial()
        File "/opt/conda/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 510, in _build_extensions_serial
          self.build_extension(ext)
        File "/opt/conda/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 264, in build_extension
          _build_ext.build_extension(self, ext)
        File "/opt/conda/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 565, in build_extension
          objects = self.compiler.compile(
                    ^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/conda/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 866, in unix_wrap_ninja_compile
          _write_ninja_file_and_compile_objects(
        File "/opt/conda/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 2223, in _write_ninja_file_and_compile_objects
          _run_ninja_build(
        File "/opt/conda/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 2614, in _run_ninja_build
          raise RuntimeError(message) from e
      RuntimeError: Error compiling objects for extension
      /opt/conda/lib/python3.11/site-packages/setuptools/_distutils/dist.py:1021: _DebuggingTips: Problem in editable installation.
      !!
      
              ********************************************************************************
              An error happened while installing `extension_cpp` in editable mode.
      
              The following steps are recommended to help debug this problem:
      
              - Try to install the project normally, without using the editable mode.
                Does the error still persist?
                (If it does, try fixing the problem before attempting the editable mode).
              - If you are using binary extensions, make sure you have all OS-level
                dependencies installed (e.g. compilers, toolchains, binary libraries, ...).
              - Try the latest version of setuptools (maybe the error was already fixed).
              - If you (or your project dependencies) are using any setuptools extension
                or customization, make sure they support the editable mode.
      
              After following the steps above, if the problem still persists and
              you think this is related to how setuptools handles editable installations,
              please submit a reproducible example
              (see https://stackoverflow.com/help/minimal-reproducible-example) to:
      
                  https://github.com/pypa/setuptools/issues
      
              See https://setuptools.pypa.io/en/latest/userguide/development_mode.html for details.
              ********************************************************************************
      
      !!
        cmd_obj.run()
      Traceback (most recent call last):
        File "/opt/conda/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 2597, in _run_ninja_build
          subprocess.run(
        File "/opt/conda/lib/python3.11/subprocess.py", line 571, in run
          raise CalledProcessError(retcode, process.args,
      subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
      
      The above exception was the direct cause of the following exception:
      
      Traceback (most recent call last):
        File "/opt/conda/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 389, in <module>
          main()
        File "/opt/conda/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 373, in main
          json_out["return_val"] = hook(**hook_input["kwargs"])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/conda/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 303, in build_editable
          return hook(wheel_directory, config_settings, metadata_directory)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/conda/lib/python3.11/site-packages/setuptools/build_meta.py", line 468, in build_editable
          return self._build_with_temp_dir(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/conda/lib/python3.11/site-packages/setuptools/build_meta.py", line 404, in _build_with_temp_dir
          self.run_setup()
        File "/opt/conda/lib/python3.11/site-packages/setuptools/build_meta.py", line 317, in run_setup
          exec(code, locals())
        File "<string>", line 75, in <module>
        File "/opt/conda/lib/python3.11/site-packages/setuptools/__init__.py", line 117, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/conda/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 186, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "/opt/conda/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 202, in run_commands
          dist.run_commands()
        File "/opt/conda/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 1002, in run_commands
          self.run_command(cmd)
        File "/opt/conda/lib/python3.11/site-packages/setuptools/dist.py", line 1104, in run_command
          super().run_command(command)
        File "/opt/conda/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
          cmd_obj.run()
        File "/opt/conda/lib/python3.11/site-packages/setuptools/command/editable_wheel.py", line 139, in run
          self._create_wheel_file(bdist_wheel)
        File "/opt/conda/lib/python3.11/site-packages/setuptools/command/editable_wheel.py", line 340, in _create_wheel_file
          files, mapping = self._run_build_commands(dist_name, unpacked, lib, tmp)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/conda/lib/python3.11/site-packages/setuptools/command/editable_wheel.py", line 263, in _run_build_commands
          self._run_build_subcommands()
        File "/opt/conda/lib/python3.11/site-packages/setuptools/command/editable_wheel.py", line 290, in _run_build_subcommands
          self.run_command(name)
        File "/opt/conda/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 357, in run_command
          self.distribution.run_command(command)
        File "/opt/conda/lib/python3.11/site-packages/setuptools/dist.py", line 1104, in run_command
          super().run_command(command)
        File "/opt/conda/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
          cmd_obj.run()
        File "/opt/conda/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 99, in run
          _build_ext.run(self)
        File "/opt/conda/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 368, in run
          self.build_extensions()
        File "/opt/conda/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1082, in build_extensions
          build_ext.build_extensions(self)
        File "/opt/conda/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 484, in build_extensions
          self._build_extensions_serial()
        File "/opt/conda/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 510, in _build_extensions_serial
          self.build_extension(ext)
        File "/opt/conda/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 264, in build_extension
          _build_ext.build_extension(self, ext)
        File "/opt/conda/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 565, in build_extension
          objects = self.compiler.compile(
                    ^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/conda/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 866, in unix_wrap_ninja_compile
          _write_ninja_file_and_compile_objects(
        File "/opt/conda/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 2223, in _write_ninja_file_and_compile_objects
          _run_ninja_build(
        File "/opt/conda/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 2614, in _run_ninja_build
          raise RuntimeError(message) from e
      RuntimeError: Error compiling objects for extension
      [end of output]

Both are still reproducible in torch 2.9.1.

[nWEIdia]

We expect "the right" CUDA version to at least recognize either SM_110 (preferred and official) and SM_101 (deprecated and undesired). If the CUDA version is not recognizing any of these two, it tells me the CUDA version is very old.
Could you please double check the "nvcc --version" that is being used? For example, is it coming from triton shipped CUDA or system installed CUDA (if system-installed, in either case, what version of CUDA is it?)

[SuibianP]

@nWEIdia - I will check the nvcc version later, but I believe it is unlikely the culprit since only the official image is used. There is also no "system CUDA" in the Docker build context if I understand its isolation correctly.

In case you missed it, a Dockerfile reproducer is actually included in the issue for full reproducibility, basing upon official PyTorch image and extension sample.

[nWEIdia]

@nWEIdia - I will check the nvcc version later, but I believe it is unlikely the culprit since only the official image is used. There is also no "system CUDA" in the Docker build context if I understand its isolation correctly.

In case you missed it, a Dockerfile reproducer is actually included in the issue for full reproducibility, basing upon official PyTorch image and extension sample.

Thanks for the reminder, I will try to reproduce it.

[nWEIdia]

I root-caused this to be indeed originating from "the renaming of sm_101 to sm_110".
The failure is not directly due to CUDA version or nvcc fatal message (using cuda 12.8 + sm_110 would be fatal and not supported). But it is because the failure happens with supported_archs missing cuda 12.8's view of THOR (Only the Old name: 10.1, not recognizing the New SM name: 11.0).
https://github.com/pytorch/pytorch/blob/v2.9.0/torch/utils/cpp_extension.py#L2407-L2410

So one workaround would be to add 10.1. I tested and then it "Successfully installed extension_cpp-0.0.1"

If you can, could you please move to 2.9.0+cu130 image? When you use 2.9.0+cu130 you can use "ENV TORCH_CUDA_ARCH_LIST=11.0"
Then: you get "Successfully installed extension_cpp-0.0.1"
This 2.9.0+cu130 is the recommended workaround.

So I would recommend moving to cu130 so that we don't get tangled with the renaming history.