Roadmap to [community]

[tpkessler]

With the last package being updated to 3.5.0 yesterday, we should think about how we can get (a part of) the packages into [community]. From our issues tracker I infer that most of the people are interested in two things:

• OpenCL,
• Machine Learning with pytorch or tensorflow.

The last point seems to be out of reach for now since it needs most of the ROCm stack which I would not consider stable enough for [community]. But maybe OpenCL could find its way in there. For this we need at least:

• hsakmt-roct
• hsa-rocr
• rocm-device-libs
• comgr

llvm-amdgpu in this case is only needed to generate rocm-device-libs. The new backend rocclr in its current form is a collection of headers and a static library, so we may ship it with rocm-opencl-runtime like it's presumably done by AMD. In contrast to other packages in this repo the four mentioned above are relatively small. This seems to make them easily maintainable.

Once we have this, we can turn to hip and try to add it to [community]. At the moment this would mean that we also need to add llvm-amdgpu to [community] which would mean that two llvm packages are in the arch repos. In my opinion we'll have to wait for upstream LLVM to catch up with AMD's extensions. Maybe with LLVM 11 we can replace llvm-amdgpu with llvm given it is compiled with AMDGPU as an additional target.

What's your opinion? I would also appreciate feedback from @ArchangeGabriel and @lordheavy, who already contributed to this repo.

[naetherm]

In my opinion we'll have to wait for upstream LLVM to catch up with AMD's extensions.

That's exactly what I thought, otherwise everything would become irritating.

[ArchangeGabriel]

Hi there,

So my comments on this: targeting OpenCL only at first seems a good idea to start. llvm-amdgpu would not be able to enter [community], so we indeed need to wait for LLVM to release a version with AMD extensions inside.

[acxz]

Just an update, python-pytorch-rocm and tensorflow-rocm build and run successfully now!

[tpkessler]

@acxz : That's great news!

we indeed need to wait for LLVM to release a version with AMD extensions inside

I configured llvm-git (LLVM 12) with the additional flag

-D LLVM_TARGETS_TO_BUILD='AMDGPU;X86'

and can build rocm-device-libs with upstream clang:

diff --git a/rocm-device-libs/PKGBUILD b/rocm-device-libs/PKGBUILD
index 9ae77c4..2cd42cf 100644
--- a/rocm-device-libs/PKGBUILD
+++ b/rocm-device-libs/PKGBUILD
@@ -13,9 +13,10 @@ sha256sums=('dce3a4ba672c4a2da4c2260ee4dc96ff6dd51877f5e7e1993cb107372a35a378')
 _dirname="$(basename "$url")-$(basename "${source[0]}" .tar.gz)"
 
 build() {
-    CC=/opt/rocm/llvm/bin/clang \
+    CC=/usr/bin/clang \
     cmake   -DCMAKE_INSTALL_PREFIX=/opt/rocm \
-            -DLLVM_DIR=/opt/rocm/llvm/lib/cmake/llvm \
+            -DLLVM_DIR=/usr/lib/cmake/llvm \
+            -DClang_DIR=/usr/lib/cmake/clang \
             "$_dirname"
     make
 }

UPDATE:
There's hope! rocm-device-libs also builds with the LLVM 11 release candidate.

[tpkessler]

How should we proceed? I suppose that AMD will release ROCm 3.6.0 next week (maybe on Monday?), so should we focus on getting 3.6.0 into [community]? LLVM 11 is scheduled for August 26, so the next step would be to open a feature request on bugs.archlinux.org in order to get AMDGPU support for the Arch llvm and clang packages?

[ArchangeGabriel]

Did you try with LLVM 11 then? If so I can definitively ask @foutrelis to build AMDGPU target for the next release.

[tpkessler]

Yes, it works with LLVM 11. I've updated my comment.

[foutrelis]

I see AMDGPU in our LLVM 10; is it not sufficient?

$ llvm-config --targets-built
AArch64 AMDGPU ARM BPF Hexagon Lanai Mips MSP430 NVPTX PowerPC RISCV Sparc SystemZ WebAssembly X86 XCore AVR

[tpkessler]

That's interesting. I always thought that one has to explicitly specify AMDGPU. The relevant code to build rocm-device-libs isn't included in LLVM 10. We have to wait for LLVM 11. But then it should work. Thanks for the information @foutrelis!

[ArchangeGabriel]

All right, then we just have to wait for LLVM 11 to be released and things can move bits by bits to [community]. I’ll be quite busy when it will land, so hopefully someone else will be interested or it will have to wait a bit.

[acxz]

Lately other projects have also been taking my attention, but @tpkessler has really stepped up in maintaining and upgrading these packages. I would nominate him, @tpkessler willing, if you guys need a helping hand or someone to maintain these packages over at [community].

Due to the complexity of these packages tho, I would like to keep this repo active, so that ROCm users on Arch Linux can easily chip in PRs, report issues, ask for help, etc. If it would be possible to keep the same format that we have right now, but just change the syncing process from the AUR to the [community] repos that would be really great.

[tpkessler]

Thank you for you kind words @acxz, I appreciate it. Of course, I'm willing to help. Our focus should be on the following packages:

• hsakmt-roct
• hsa-rocr
• rocm-device-libs
• comgr
• rocclr

They provide everything that is needed to get OpenCL (and HIP ?) started.

[ArchangeGabriel]

LLVM 11 has been added to [staging]. Please try this first list again it and we will see what can we do. ;)

@acxz Mid-term goal for Arch is to move the packages tree to git + GitLab to be able to accept PR from the community. :) But meanwhile we can keep this repo here for sure!

[tpkessler]

I can confirm that rocm-device-libs successfully builds with LLVM 11 from [staging]. I'll rebuild the rest of packages later and will run some tests to see if we get functioning packages.

[tpkessler]

comgr as of version 3.8.0 does not build with LLVM 11 because of an incompatible usage of LLVM-internal data types. This is fixed with the newest git version which, however, includes code for BigNavi that is not yet part of LLVM 11. Maybe this will change with the LLVM 11.1 release?

[lordheavy]

So now, what packages could we push to community ?

[tpkessler]

All but two packages depend on rocm-device-libs for which we need llvm-amdgpu since the latter is based on LLVM 12 and already includes support for AMD GPUs that LLVM 11.1 is lacking.
In order to push ROCm to community we either have to add llvm-amdgpu to community or wait for upstream LLVM to catch up.

[tpkessler]

ROCm 4.2 has been released for a while and the packages are in a good shape. For OpenCL (the bare minimum for ROCm) we need

• llvm-amdgpu
• hsakmt-roct
• rocm-device-libs
• hsa-rocr
• rocm-cmake
• comgr
• rocclr
• rocm-opencl-runtime
• rocm-clang-ocl

It is crucial to include llvm-amdgpu as AMD pushes changes to their fork faster than they are excepted to mainline LLVM. This often means that llvm-amdgpu includes support for the newest architectures that has not been merged into upstream yet.
What's your opinion on this @lordheavy , @ArchangeGabriel ?

The remaining packages are small (They amount to 124 MiB with comgr having a size of 112 MiB) and are built by the usual cmake --> make --> make install chain.

For HIP we additionally need

• rocminfo
• hip-rocclr

Useful utility packages include

• rocm-dbgapi
• rocm-gdb
• rocm-smi-lib64
• rocprofiler
• roctracer

Once we have that, we can include the different libraries from rocm-lib.

[ArchangeGabriel]

Sorry for the delay, apparently missed the notification.

I still don’t think having llvm-amdgpu in [community] is possible/a good idea. We generally don’t allow fork in addition to mainstream project, so it would be better if we could use standard llvm, even if that means latest archs are not supported. I’ll ask internally for opinions on this though.

[tpkessler]

Hey @ArchangeGabriel, thank you for coming back to this issue!

We generally don’t allow fork in addition to mainstream project

I see. That's indeed a problem.

it would be better if we could use standard llvm

Yes, I agree. Currently, this is not possible as many of ROCm's (HIP or OpenMP offloading) features depend on llvm-amdgpu.

[sandboiii]

Hey, LLVM recently updated to 13.0.0. Are there still any issues with building the packages?

[tpkessler]

Not for rocm-device-libs and hsa-rocr, but comgr-4.5.2does not build with upstream llvm, see #640. This may be fixed with the next ROCm release.

[darkbasic]

Is it fixed in 5.0.1?

[naetherm]

As rocm-llvm is still a dependency of comgr, I don't think so:
https://aur.archlinux.org/packages/comgr

[acxz]

Wazzup @naetherm its been a while haha! Hope u doing well!
Yeah I just tried compiling comgr-5.0.1 with llvm-13.0.1 to no avail.

[lordheavy]

Finally blender supports rocm, it should be great for one of you to apply as Trusted User, so archlinuix packages will be available in our repo. I think the priority is to have rocm merged, even if it results in duplicated LLVM - something we can't avoid for now. Feel free to apply and/or comment OFC.

[lordheavy]

Sure, I can be your sponsor - with pleasure - it's the moment to get rid of this !

[acxz]

This is wonderful! I am quite overjoyed right now!

[svenstaro]

@tpkessler Good stuff! Looks like you're already quite popular. :) Let's hop over to emails and get your application written.

[darkbasic]

@tpkessler is there any news on this?

[tpkessler]

Yes, there are! I was elected as a Trusted User last week, see my application on the mailing list. I'm currently in the onboarding phase and hope to contribute to Arch (in particular ROCm in [community]) soon.