Add gcc +amdgcn

[wyphan]

• This is done by following this guide on the official GCC wiki (and adapting as necessary): GCC offloading.
• I made some code that was specific to +nvptx variant common to both +nvptx and +amdgcn.
• Newlib SHA256 sums manually computed after verifying with the provided SHA512 sums.
• The GCC +amdgcn build only needs LLVM utils, and not the clang compiler itself. I guess this causes LLVM to be a link time dependency, even though the utils are copied over to the GCC install dir?
• The lib/lib64 directory is RPATH-ed everytime -fopenmp or -fopenacc is active so the produced executable uses the correct libgomp.so.

[wyphan]

@spackbot fix style

[spackbot-app]

Let me see if I can fix that for you!

[spackbot-app]

I was able to run spack style --fix for you!

spack style --fix

==> Running style checks on spack
  selected: isort, black, flake8, mypy
==> Modified files
  var/spack/repos/builtin/packages/gcc/package.py
==> Running isort checks
  isort checks were clean
==> Running black checks
reformatted var/spack/repos/builtin/packages/gcc/package.py
All done! ✨ 🍰 ✨
1 file reformatted.
  black checks were clean
==> Running flake8 checks
  flake8 checks were clean
==> Running mypy checks
Success: no issues found in 576 source files
  mypy checks were clean
==> spack style checks were clean

Keep in mind that I cannot fix your flake8 or mypy errors, so if you have any you'll need to fix them and update the pull request. If I was able to push to your branch, if you make further changes you will need to pull from your updated branch before pushing again.

I've updated the branch with style fixes.

[wyphan]

I just built gcc@12.2.0 +amdgcn locally on my workstation. It finishes building successfully!
The problem is when I test this Spack-built GCC with an OpenACC test code, I get the following error message:

wyp@basecamp:~/work/testcodes/f90-acc-ddot$ gfortran -fopenacc -foffload=amdgcn-unknown-amdhsa="-march=gfx900" ddot.f90 
as: unrecognized option '-triple=amdgcn--amdhsa'
mkoffload: fatal error: x86_64-pc-linux-gnu-accel-amdgcn-unknown-amdhsa-gcc returned 1 exit status
compilation terminated.
lto-wrapper: fatal error: /home/wyp/work/spack/opt/spack/linux-ubuntu20.04-zen2/gcc-12.2.0/gcc-12.2.0-w7lclfarefmge3uegn2a5vw37bnwhwto/libexec/gcc/x86_64-pc-linux-gnu/12.2.0//accel/amdgcn-unknown-amdhsa/mkoffload returned 1 exit status
compilation terminated.
/usr/bin/ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status

If it helps, the supported targets from llvm-mc, which is copied as as into GCC offload directory, are:

yp@basecamp:~/work/spack/opt/spack/linux-ubuntu20.04-zen2/gcc-12.2.0/llvm-13.0.1-6zvbqlbev3wiqihvz2god7pzcptnoxu3/bin$ llvm-mc -version
LLVM (http://llvm.org/):
  LLVM version 13.0.1
  Optimized build.
  Default target: x86_64-unknown-linux-gnu
  Host CPU: znver2

  Registered Targets:
    amdgcn - AMD GCN GPUs
    r600   - AMD GPUs HD2XXX-HD6XXX
    x86    - 32-bit X86: Pentium-Pro and above
    x86-64 - 64-bit X86: EM64T and AMD64

@michaelkuhn Do you know how I can fix this?

[wyphan]

After some correspondence with the devs on the GCC mailing list, I think I figured it out, which has been implemented in commit 9fda94b. I just successfully compiled and ran both an OpenACC Fortran and OpenMP target offload C code. The catch is that I needed to export the Spack prefix's lib64 directory into LD_LIBRARY_PATH such that it loads the proper libgomp.so instead of using the system one.

[spackbot-app]

Let me see if I can fix that for you!

[spackbot-app]

I was able to run spack style --fix for you!

spack style --fix

==> Running style checks on spack
  selected: isort, black, flake8, mypy
==> Modified files
  var/spack/repos/builtin/packages/gcc/package.py
==> Running isort checks
  isort checks were clean
==> Running black checks
reformatted var/spack/repos/builtin/packages/gcc/package.py
All done! ✨ 🍰 ✨
1 file reformatted.
  black checks were clean
==> Running flake8 checks
  flake8 checks were clean
==> Running mypy checks
Success: no issues found in 588 source files
  mypy checks were clean
==> spack style checks were clean

Keep in mind that I cannot fix your flake8 or mypy errors, so if you have any you'll need to fix them and update the pull request. If I was able to push to your branch, if you make further changes you will need to pull from your updated branch before pushing again.

I've updated the branch with style fixes.

[pranav-sivaraman]

@wyphan I fixed the earlier issue I was having. The last time I installed this you did not revert the llvm util path. So I just manually moved it in the install location and was able to compile it.

Pulling in the new changes, the new issue I am facing is that find_package(OpenACC) seems to not work at all?

This is for BabelStream on the main branch. The cmake command I used was cmake -S . -B build -DMODEL=acc -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++

[wyphan]

Pulling in the new changes, the new issue I am facing is that find_package(OpenACC) seems to not work at all?

This is for BabelStream on the main branch. The cmake command I used was cmake -S . -B build -DMODEL=acc -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++

Hmm, perhaps try adding -DOpenACC_CXX_FLAGS=-fopenacc?

I just tested with a Fortran OpenACC code from my test codes suite at https://github.com/wyphan/testcodes/tree/master/f90-acc-ddot and it works just fine:

wyp@basecamp:~/work/testcodes/f90-acc-ddot$ gfortran -g -O2 -fopenacc -foffload="-g -O2" -foffload=amdgcn-amdhsa="-march=gfx900" ddot.f90  -o ddot.gcc.x
wyp@basecamp:~/work/testcodes/f90-acc-ddot$ ./ddot.gcc.x 1000000
 Using n =      1000000
 Success! Result =    6.6666766666625510E+017
Initialization took    0.005 seconds.
Computation took    0.001 seconds.

[pranav-sivaraman]

@wyphan Still having the same issue after pulling in the new changes. I tried compiling your Fortran OpenACC code and was getting the previous issue.

If I do a which as after doing spack load gcc. It still picks up the as installed on the system not the as provided by LLVM.

If you change where the LLVM utilities are installed from the libexec folder to the amdgcn-amdhsa/bin/ folder gcc will pick up the right as

After doing the above I can compile and run without needing to mess with LD_LIBRARY_PATH

[wyphan]

@spackbot fix style

[spackbot-app]

Let me see if I can fix that for you!

[spackbot-app]

I was able to run spack style --fix for you!

spack style --fix

==> Running style checks on spack
  selected: isort, black, flake8, mypy
==> Modified files
  var/spack/repos/builtin/packages/gcc/package.py
==> Running isort checks
  isort checks were clean
==> Running black checks
reformatted var/spack/repos/builtin/packages/gcc/package.py
All done! ✨ 🍰 ✨
1 file reformatted.
  black checks were clean
==> Running flake8 checks
  flake8 checks were clean
==> Running mypy checks
Success: no issues found in 588 source files
  mypy checks were clean
==> spack style checks were clean

Keep in mind that I cannot fix your flake8 or mypy errors, so if you have any you'll need to fix them and update the pull request. If I was able to push to your branch, if you make further changes you will need to pull from your updated branch before pushing again.

I've updated the branch with style fixes.

[wyphan]

@pranav-sivaraman Can you test again? Honestly I'm unsure why the strace output after commit 9be9a2b looks like this:

2037932 stat("/home/wyp/work/spack/opt/spack/linux-ubuntu20.04-zen2/gcc-9.4.0/gcc-13.2.0-6jy7jhczbpoyilf3ufysupwlnrz2kgrg/libexec/gcc/x86_64-pc-linux-gnu/13.2.0/accel/amdgcn-amdhsa/as", 0x7fff8d75c390) = -1 ENOENT (No such file or directory)
2037932 stat("/home/wyp/work/spack/opt/spack/linux-ubuntu20.04-zen2/gcc-9.4.0/gcc-13.2.0-6jy7jhczbpoyilf3ufysupwlnrz2kgrg/libexec/gcc/x86_64-pc-linux-gnu/13.2.0/accel/amdgcn-amdhsa/as", 0x7fff8d75c390) = -1 ENOENT (No such file or directory)
2037932 stat("/home/wyp/work/spack/opt/spack/linux-ubuntu20.04-zen2/gcc-9.4.0/gcc-13.2.0-6jy7jhczbpoyilf3ufysupwlnrz2kgrg/libexec/gcc/amdgcn-amdhsa/as", 0x7fff8d75c390) = -1 ENOENT (No such file or directory)
2037932 stat("/home/wyp/work/spack/opt/spack/linux-ubuntu20.04-zen2/gcc-9.4.0/gcc-13.2.0-6jy7jhczbpoyilf3ufysupwlnrz2kgrg/lib/gcc/x86_64-pc-linux-gnu/13.2.0/accel/amdgcn-amdhsa/as", 0x7fff8d75c390) = -1 ENOENT (No such file or directory)
2037932 stat("/home/wyp/work/spack/opt/spack/linux-ubuntu20.04-zen2/gcc-9.4.0/gcc-13.2.0-6jy7jhczbpoyilf3ufysupwlnrz2kgrg/lib/gcc/amdgcn-amdhsa/as", 0x7fff8d75c390) = -1 ENOENT (No such file or directory)
2037932 stat("/home/wyp/work/spack/opt/spack/linux-ubuntu20.04-zen2/gcc-9.4.0/gcc-13.2.0-6jy7jhczbpoyilf3ufysupwlnrz2kgrg/lib/gcc/x86_64-pc-linux-gnu/13.2.0/accel/amdgcn-amdhsa/../../../../../../amdgcn-amdhsa/bin/x86_64-pc-linux-gnu/13.2.0/accel/amdgcn-amdhsa/as", 0x7fff8d75c390) = -1 ENOENT (No such file or directory)
2037932 stat("/home/wyp/work/spack/opt/spack/linux-ubuntu20.04-zen2/gcc-9.4.0/gcc-13.2.0-6jy7jhczbpoyilf3ufysupwlnrz2kgrg/lib/gcc/x86_64-pc-linux-gnu/13.2.0/accel/amdgcn-amdhsa/../../../../../../amdgcn-amdhsa/bin/as", {st_mode=S_IFREG|0755, st_size=29256600, ...}) = 0
2037932 access("/home/wyp/work/spack/opt/spack/linux-ubuntu20.04-zen2/gcc-9.4.0/gcc-13.2.0-6jy7jhczbpoyilf3ufysupwlnrz2kgrg/lib/gcc/x86_64-pc-linux-gnu/13.2.0/accel/amdgcn-amdhsa/../../../../../../amdgcn-amdhsa/bin/as", X_OK) = 0

Notice that it tried to access the libexec tree first.