Skip to content

ciflow/trunk/194942

ciflow/trunk/194942: Derive the inductor benchmark wheel index from the toolkit version
@tinglvv tinglvv tagged this 10 Sep 00:13 
install_inductor_benchmark_deps.sh installs a stable torch purely to satisfy
TorchBench's install check, hardcoding the cu130 index for every CUDA 13.x
image. That torch is uninstalled afterwards, but its nvidia-* dependency wheels
are not: the cleanup names only nccl and cudnn, so a CUDA 13.0
libcublasLt.so.13 is left behind in site-packages.

The leftover is not inert. _load_global_deps() preloads component wheels found
in site-packages before torch._C is imported, deliberately so they win
libtorch_cpu's DT_NEEDED soname lookups over a system copy. On a CUDA 13.4
image the preview toolkit headers satisfy the CUDA_VERSION >= 13030 guard in
CublasLtUtils.h, so libtorch_cuda.so references
cublasLtGroupedMatrixLayoutCreate, which a 13.0 cuBLAS does not export. The
dlopen fails, surfacing as a message-less ImportError from torchvision's
setup.py -- the only place a 13.4 build job imports torch.

Only 13.4 is affected because it is the sole configuration above the 13030
threshold; 13.2 and below preprocess the call away, so the same mismatched
wheel has been harmless there since it was introduced.

Deriving the index from DESIRED_CUDA keeps the benchmark wheels on the same
CUDA minor as the toolkit. Preview toolkits have no stable index yet, so the
index is probed and falls back to nightly. Probing rather than hardcoding which
versions are preview means no edit is needed once cu134 publishes. The
non-version branch is kept because INDUCTOR_BENCHMARKS is also set on images
where DESIRED_CUDA is unset, which a bare derivation would break.

Adding nvidia-cublas to the uninstall list was the alternative. It fixes this
symptom but leaves an allowlist that will miss the next orphaned wheel, whereas
matching the index removes the reason a mismatch exists.

The cu130 wheel's cuBLAS was not inspected directly; the diagnosis rests on the
version guard and the documented preload behaviour, so the CI job is the real
confirmation.

Test Plan:

Syntax and version derivation were checked locally:

    bash -n .ci/docker/common/install_inductor_benchmark_deps.sh

which passes, and the derivation maps 13.4.0/13.2.1/13.0.2/12.8.1 to
cu134/cu132/cu130/cu128, with unset and non-version values falling back to
cu128.

Index availability was confirmed with:

    curl -s -o /dev/null -w "%{http_code}" https://download.pytorch.org/whl/cu132/torch/
    curl -s -o /dev/null -w "%{http_code}" https://download.pytorch.org/whl/cu134/torch/
    curl -s -o /dev/null -w "%{http_code}" https://download.pytorch.org/whl/nightly/cu134/torch/

returning 200, 403 and 200 respectively.

Real verification is the inductor-build-cuda134 job, which should receive a
13.4-matching cuBLAS and import torch successfully. lintrunner is not installed
in this checkout, so the shell lint was not run.

Authored with assistance from an AI coding assistant (Claude).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Assets 2
Loading