[Multi-arch][Issue] Linux python missing gfx942/gfx950 device extras (Windows is generating conflicting `rocm` python sdists)

[ScottTodd]

Symptoms

Observed on both

• nightly: https://rocm.nightlies.amd.com/whl-multi-arch/
• stable: https://repo.amd.com/rocm/whl-multi-arch/

Running

pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ "rocm[device-all]"

pip freeze
# rocm-sdk-device-gfx1010==7.14.0a2026051
# rocm-sdk-device-gfx1100==7.14.0a2026051
...
# MISSING rocm-sdk-device-gfx942
# MISSING rocm-sdk-device-gfx950

on Linux does not download the rocm-sdk-device-gfx942 or rocm-sdk-device-gfx950 packages and may fail at runtime with missing data errors (or worse, as many of the libraries assume that their files exist and don't handle this failure mode well)

Explicitly installing the device extra does not work either:

pip install --index-url https://rocm.nightlies.amd.com/whl-multi-arch/ rocm[device-gfx950] --dry-run

WARNING: rocm 7.14.0a20260519 does not provide the extra 'device-gfx950'
Collecting rocm-sdk-core==7.14.0a20260519 (from rocm[device-gfx950])
  Downloading rocm_sdk_core-7.14.0a20260519-py3-none-linux_x86_64.whl (410.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 410.9/410.9 MB 40.6 MB/s  0:00:09
Would install rocm-7.14.0a20260519 rocm-sdk-core-7.14.0a20260519

Depending on which platform uploaded the most recent sdist, Windows device-all may attempt to install those packages and emit a warning when it fails.

Workarounds

The rocm-sdk-device-gfx942 and rocm-sdk-device-gfx950 packages can still be installed directly.

pip install --index-url https://rocm.nightlies.amd.com/whl-multi-arch/ rocm rocm-sdk-device-gfx950 --dry-run

Would install rocm-7.14.0a20260519 rocm-sdk-core-7.14.0a20260519 rocm-sdk-device-gfx950-7.14.0a20260519 rocm-sdk-libraries-7.14.0a20260519

The torch packages are platform-specific with functioning extras that directly depend on the corresponding rocm packages, so this works:

pip install --index-url https://rocm.nightlies.amd.com/whl-multi-arch/ torch[device-gfx950] --dry-run

Would install Jinja2-3.1.6 MarkupSafe-3.0.3 amd-torch-device-gfx950-2.12.0+rocm7.14.0a20260519 filelock-3.29.0 fsspec-2026.4.0 mpmath-1.3.0 networkx-3.6.1 rocm-7.14.0a20260519 rocm-bootstrap-0.1.0 rocm-sdk-core-7.14.0a20260519 rocm-sdk-device-gfx950-7.14.0a20260519 rocm-sdk-libraries-7.14.0a20260519 setuptools-81.0.0 sympy-1.14.0 torch-2.12.0+rocm7.14.0a20260519 triton-3.7.0+git52687acf.rocm7.14.0a20260519 typing_extensions-4.15.0

The issue

Metadata in rocm-7.13.0.tar.gz --> rocm-7.13.0/PKG-INFO is of the form:

 Metadata-Version: 2.4
 Name: rocm
 Version: 7.13.0
 Requires-Dist: rocm==7.13.0
 Requires-Dist: rocm-sdk-core==7.13.0
 Provides-Extra: libraries
 Requires-Dist: rocm-sdk-libraries==7.13.0; extra == "libraries"
 Provides-Extra: device
 Requires-Dist: rocm-sdk-device-gfx1010==7.13.0; extra == "device"
 Provides-Extra: devel
 Requires-Dist: rocm-sdk-devel==7.13.0; extra == "devel"
 Provides-Extra: profiler
 Requires-Dist: rocm-profiler==7.13.0; extra == "profiler"
 Provides-Extra: device-gfx1010
 Requires-Dist: rocm-sdk-device-gfx1010==7.13.0; extra == "device-gfx1010"
 Provides-Extra: device-gfx1011
 Requires-Dist: rocm-sdk-device-gfx1011==7.13.0; extra == "device-gfx1011"
 Provides-Extra: device-gfx1012
 Requires-Dist: rocm-sdk-device-gfx1012==7.13.0; extra == "device-gfx1012"
 Provides-Extra: device-gfx1030
 Requires-Dist: rocm-sdk-device-gfx1030==7.13.0; extra == "device-gfx1030"
 Provides-Extra: device-gfx1031
 Requires-Dist: rocm-sdk-device-gfx1031==7.13.0; extra == "device-gfx1031"
 Provides-Extra: device-gfx1032
 Requires-Dist: rocm-sdk-device-gfx1032==7.13.0; extra == "device-gfx1032"
 Provides-Extra: device-gfx1033
 Requires-Dist: rocm-sdk-device-gfx1033==7.13.0; extra == "device-gfx1033"
 Provides-Extra: device-gfx1034
 Requires-Dist: rocm-sdk-device-gfx1034==7.13.0; extra == "device-gfx1034"
 Provides-Extra: device-gfx1035
 Requires-Dist: rocm-sdk-device-gfx1035==7.13.0; extra == "device-gfx1035"
 Provides-Extra: device-gfx1036
 Requires-Dist: rocm-sdk-device-gfx1036==7.13.0; extra == "device-gfx1036"
 Provides-Extra: device-gfx1100
 Requires-Dist: rocm-sdk-device-gfx1100==7.13.0; extra == "device-gfx1100"
 Provides-Extra: device-gfx1101
 Requires-Dist: rocm-sdk-device-gfx1101==7.13.0; extra == "device-gfx1101"
 Provides-Extra: device-gfx1102
 Requires-Dist: rocm-sdk-device-gfx1102==7.13.0; extra == "device-gfx1102"
 Provides-Extra: device-gfx1103
 Requires-Dist: rocm-sdk-device-gfx1103==7.13.0; extra == "device-gfx1103"
 Provides-Extra: device-gfx1150
 Requires-Dist: rocm-sdk-device-gfx1150==7.13.0; extra == "device-gfx1150"
 Provides-Extra: device-gfx1151
 Requires-Dist: rocm-sdk-device-gfx1151==7.13.0; extra == "device-gfx1151"
 Provides-Extra: device-gfx1152
 Requires-Dist: rocm-sdk-device-gfx1152==7.13.0; extra == "device-gfx1152"
 Provides-Extra: device-gfx1200
 Requires-Dist: rocm-sdk-device-gfx1200==7.13.0; extra == "device-gfx1200"
 Provides-Extra: device-gfx1201
 Requires-Dist: rocm-sdk-device-gfx1201==7.13.0; extra == "device-gfx1201"
-Provides-Extra: device-gfx942
-Requires-Dist: rocm-sdk-device-gfx942==7.13.0; extra == "device-gfx942"
-Provides-Extra: device-gfx950
-Requires-Dist: rocm-sdk-device-gfx950==7.13.0; extra == "device-gfx950"
 Provides-Extra: device-all
 Requires-Dist: rocm-sdk-device-gfx1010==7.13.0; extra == "device-all"
 Requires-Dist: rocm-sdk-device-gfx1011==7.13.0; extra == "device-all"
 Requires-Dist: rocm-sdk-device-gfx1012==7.13.0; extra == "device-all"
 Requires-Dist: rocm-sdk-device-gfx1030==7.13.0; extra == "device-all"
 Requires-Dist: rocm-sdk-device-gfx1031==7.13.0; extra == "device-all"
 Requires-Dist: rocm-sdk-device-gfx1032==7.13.0; extra == "device-all"
 Requires-Dist: rocm-sdk-device-gfx1033==7.13.0; extra == "device-all"
 Requires-Dist: rocm-sdk-device-gfx1034==7.13.0; extra == "device-all"
 Requires-Dist: rocm-sdk-device-gfx1035==7.13.0; extra == "device-all"
 Requires-Dist: rocm-sdk-device-gfx1036==7.13.0; extra == "device-all"
 Requires-Dist: rocm-sdk-device-gfx1100==7.13.0; extra == "device-all"
 Requires-Dist: rocm-sdk-device-gfx1101==7.13.0; extra == "device-all"
 Requires-Dist: rocm-sdk-device-gfx1102==7.13.0; extra == "device-all"
 Requires-Dist: rocm-sdk-device-gfx1103==7.13.0; extra == "device-all"
 Requires-Dist: rocm-sdk-device-gfx1150==7.13.0; extra == "device-all"
 Requires-Dist: rocm-sdk-device-gfx1151==7.13.0; extra == "device-all"
 Requires-Dist: rocm-sdk-device-gfx1152==7.13.0; extra == "device-all"
 Requires-Dist: rocm-sdk-device-gfx1200==7.13.0; extra == "device-all"
 Requires-Dist: rocm-sdk-device-gfx1201==7.13.0; extra == "device-all"
-Requires-Dist: rocm-sdk-device-gfx942==7.13.0; extra == "device-all"
-Requires-Dist: rocm-sdk-device-gfx950==7.13.0; extra == "device-all"
 Dynamic: provides-extra
 Dynamic: requires-dist

Linux and Windows both generate this sdist with the exact same filename uploaded to the exact same location, but these platforms have different sets of available extras.

Looking back through nightly releases, some packages contain Linux-only devices and some contain just the Windows devices. There is likely a race condition where both Linux and Windows release workflows are uploading the same platform-neutral sdist packages.

Possible solutions

1. Add per-platform extras by having both Linux and Windows jobs generate identical rocm sdist metadata from a common cross-platform target list not just the per-platform target lists
2. Rework the rocm package to produce py3-none-linux_x86_64.whl and py3-none-win_amd64.whl packages like we already do for rocm-sdk-core and other packages
   • Note that per https://github.com/ROCm/TheRock/blob/main/docs/packaging/python_packaging.md#general-design this package was designed to be an sdist originally:

     Selector package: The rocm package is built as an sdist so that it is evaluated at the point of install, allowing it to perform some selection logic to detect dependencies and needs. Most other packages are installed by asking to install extras of this one (i.e. rocm[libraries], etc).

References:

• https://github.com/ROCm/TheRock/tree/main/build_tools/packaging/python/templates/rocm
• https://github.com/ROCm/TheRock/blob/main/build_tools/build_python_packages.py
• https://github.com/ROCm/TheRock/blob/main/docs/packaging/python_packaging.md
• https://github.com/ROCm/TheRock/blob/main/.github/workflows/build_portable_linux_python_packages.yml
• https://github.com/ROCm/TheRock/blob/main/.github/workflows/build_windows_python_packages.yml
• https://github.com/ROCm/TheRock/blob/main/.github/workflows/multi_arch_release.yml (note linux_amdgpu_families, windows_amdgpu_families which can differ)

[ScottTodd]

If we convert the rocm package from an sdist to a wheel, we'd lose the ability for pip install rocm[device] to auto-detect at build/install time with this code:

• TheRock/build_tools/packaging/python/templates/rocm/setup.py

  Lines 79 to 90 in 48e5305

  	# Drop the generic 'device' extra when target resolution would silently fall
  	# back to DEFAULT_TARGET_FAMILY - e.g. kpack-split CI installs on GPU-less
  	# runners where offload-arch is not yet available. Callers can still name a
  	# specific ISA via the per-target extras emitted above. Keep the generic
  	# 'device' extra when target resolution succeeds (normal install on a GPU
  	# machine).
  	device_entry = dist_info.ALL_PACKAGES.get("device")
  	if device_entry and device_entry.is_target_specific:
  	try:
  	dist_info.determine_target_family()
  	except Exception:
  	EXTRAS_REQUIRE.pop("device", None)

• TheRock/build_tools/packaging/python/templates/rocm/src/rocm_sdk/_dist_info.py

  Lines 164 to 169 in 48e5305

  	# Resolve the build target family. This consults a list of things in increasing
  	# order of specificity:
  	# 1. "ROCM_SDK_TARGET_FAMILY" environment variable
  	# 2. Dynamically discovered/most salient target family on the actual system
  	# 3. dist_info.DEFAULT_TARGET_FAMILY
  	def determine_target_family() -> str:

We could argue that the multi-arch index (especially with #5289 under discussion) is a suitable substitute for that, and we haven't yet advertised pip install rocm[device] in any documentation like https://github.com/ROCm/TheRock/blob/main/RELEASES.md (only rocm[device-all] and specific rocm[device-gfx950])

[ScottTodd]

We have several testing gaps, though actually catching this will be tricky due to the race condition with windows and linux jobs overwriting at different times as jobs start/complete up to hours apart.

I manually triggered a test workflow run on gfx950 from a nightly release: https://github.com/ROCm/TheRock/actions/runs/26132646536/job/76860876521

++ Exec [/__w/TheRock/TheRock]$ /__w/TheRock/TheRock/.venv/bin/python -m pip install --index-url=https://rocm.nightlies.amd.com/whl-multi-arch/ 'rocm[libraries,device-gfx950]==7.14.0a20260519'
WARNING: The directory '/github/home/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
Looking in indexes: https://rocm.nightlies.amd.com/whl-multi-arch/
Collecting rocm==7.14.0a20260519 (from rocm[device-gfx950,libraries]==7.14.0a20260519)
  Downloading rocm-7.14.0a20260519.tar.gz (19 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting rocm-sdk-core==7.14.0a20260519 (from rocm==7.14.0a20260519->rocm[device-gfx950,libraries]==7.14.0a20260519)
  Downloading rocm_sdk_core-7.14.0a20260519-py3-none-linux_x86_64.whl (410.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 410.9/410.9 MB 88.0 MB/s  0:00:04
WARNING: rocm 7.14.0a20260519 does not provide the extra 'device-gfx950'

Install warning but no test failures, rocm-sdk test and the rest of the test workflow completed "successfully".

[ScottTodd]

PyTorch test run at https://github.com/ROCm/TheRock/actions/runs/26133685690/job/76864057639 is a bit encouraging:

++ Exec [/__w/TheRock/TheRock]$ /__w/TheRock/TheRock/.venv/bin/python -m pip install --index-url=https://rocm.nightlies.amd.com/whl-staging-multi-arch/ 'torch[device-gfx950]==2.12.0+rocm7.14.0a20260519'
...

Collecting amd-torch-device-gfx950==2.12.0+rocm7.14.0a20260519 (from torch[device-gfx950]==2.12.0+rocm7.14.0a20260519)
Collecting rocm-sdk-device-gfx950==7.14.0a20260519 (from amd-torch-device-gfx950==2.12.0+rocm7.14.0a20260519->torch[device-gfx950]==2.12.0+rocm7.14.0a20260519)

amd-torch-device-gfx950==2.12.0+rocm7.14.0a20260519
rocm==7.14.0a20260519
rocm-bootstrap==0.1.0
rocm-sdk-core==7.14.0a20260519
rocm-sdk-device-gfx950==7.14.0a20260519
rocm-sdk-libraries==7.14.0a20260519
torch==2.12.0+rocm7.14.0a20260519
triton==3.7.0+git52687acf.rocm7.14.0a20260519

So at least the torch packages directly depend on rocm-sdk-device-gfx950 instead of going through the rocm[device-gfx950] extra.

(That run did end up failing during smoketests though, filed #5355 for that)

[marbre]

PyTorch test run at https://github.com/ROCm/TheRock/actions/runs/26133685690/job/76864057639 is a bit encouraging:
[..]

So at least the torch packages directly depend on rocm-sdk-device-gfx950 instead of going through the rocm[device-gfx950] extra.

(That run did end up failing during smoketests though, filed #5355 for that)

Yes, when splitting torch, the device extra created pulls in the amd-troch-device-gfx* which itself depends on rocm-sdk-device-*:

TheRock/.github/workflows/multi_arch_release_linux_pytorch_wheels.yml

Line 289 in ca3049e

--device-requires-dist "rocm-sdk-device-@GFXARCH@ == ${ROCM_SDK_VERSION}" \