RFC0008 Multi Arch Packaging Discussion

[stellaraccident]

Discussion about the multi-arch packaging RFC.

[trixirt]

Fedora did something similar, splitting packages based of gpu families, because of library size problems before the --offload-compress feature. Splitting the packages, required the users of those packages, other packages in the distro, to also be split greatly adding to the developer burden and confusing the end user on which package they should download and no clear handling of corner cases like multiple gpu families in the same system or user swapping cards after an install.

My greatest concern is if components cannot be rebuilt as they have been with a user settable list of gpu targets as they have been this will break all of the distros.

How would a third party, who wants to support all the gpus, build against the devel versions of these libraries without having to split their applications?

How will a user know which packages to download ? without rocm-info or similar on their system even if they were knowledgeable, it would be difficult. There is no mechanism that i know of in dnf/zypper/apt that queries the system hardware to pick gpu specific packages.

How are the gpu corner cases handled?

Using generic targets, like composable_kernel is doing, would also solve some of these issues. ex/ gfx11-generic would replace all gfx1100,gfx1101,gfx1102 etc. Is there a plan to switch to generics in the future ?

[stellaraccident]

The "UI" for meta packages is still being worked out but basically in the first version, it will basically operate much like it does today. If you apt install rocm, you will get all the arches. But underneath each library package will be split into a host package and N device family packages. Most applications will depend on the host package.

Then we will re-train users with additional meta packages and tools to do more focused installs of just the device packages that they need.

The key thing this is doing that priors did not is that device package variants can exist side by side and are additive. If you have multiple GPUs in the system, you would need the device packages for each. Host side remains the same.

Switching to generics is orthogonal and will improve the ergonomics. The device family packages may contain discrete device libraries for each arch, or as migration happens, they may just contain a generic device library (or even SPIR-V, etc), depending on the library. But the package structure stays the same. Just what is in it may get more efficient.

[superm1]

There is a concept of adding metadata to a package to indicate what hardware it supports via modaliases.
This is something that is utilized by the ubuntu-drivers package

The high level idea is you run the tool to say are there any packages that exist in repositories which enable hardware on my system. Then you're prompted to install such packages.

This doesn't perfectly match to how ROCm is organized though because it's based upon anything in a modalias which notably means you need approximately one bazillion PCI IDs and need to map those out to ISAs.

BUT - if we want to go down this path and we're building our own packages, why not encode the metadata in the package in a new AMD specific field? It's literally variable substitution in a debian field you can parse from the apt metadata.

So I could envision a world that we have a field that is AMD-GFX-ISA which populates exactly which packages go with which ISA(s).
Then we have a much simpler tool than rocminfo which maps out from sysfs what hardware you have and what packages are needed.

We have it all in /sys/class/kfd/kfd/topology/nodes. You just read the properties file and fetch out gfx_target_version. The semantics of how the tool works (does it install packages, does it tell you everything needed for the system is installed or does it do something different) is totally up for discussion. The important part is it could fit into the equation that an ISV can use to make sure that you have all the right packages that support the hardware on your system.