AOMP is a scripted build of LLVM and supporting software. It has support for OpenMP target offload on amdgcn GPUs. This is the AOMP developer README stored at:
https://github.com/ROCm-Developer-Tools/aomp/blob/master/bin/README.md
The AOMP compiler supports OpenMP, clang-hip, clang-cuda, device OpenCL, and the offline kernel compilation tool called cloc. It contains a recent version of the AMD Lightning compiler (llvm amdgcn backend) and the llvm nvptx backend. Except for clang-cuda, this compiler works for both Nvidia and AMD Radeon GPUs.
This bin directory contains scripts to build and install AOMP from source.
clone_aomp.sh - A script to make sure the necessary repos are up to date.
See below for a list of these source repositories.
build_aomp.sh - This is the master build script. It runs all the
component build scripts in the correct order.
build_roct.sh - Builds the hsa thunk library.
build_rocr.sh - Builds the ROCm runtime.
build_project.sh - Builds llvm, lld, and clang components.
build_libdevice.sh - Builds the rocdl device bc libraries from rocm-device-libs.
build_comgr.sh - Builds the code object manager (needs rocm-device-libs).
build_rocminfo.sh - Builds the rocminfo utilities to support hip.
build_vdi.sh - Builds the hip interoperability layer ROCclr.
build_hipvdi.sh - Builds the hip host runtime using ROCclr.
build_extras.sh - Builds hostcall, libm, and utils all in aomp-extras repo.
build_openmp.sh - Builds the OpenMP libraries and device RTL.
build_pgmath.sh - Builds the pgmath support for flang.
build_flang.sh - Builds flang for aomp.
build_flang_runtime.sh - Builds the flang runtime for aomp.
build_ocl.sh - Builds OpenCL runtime.
build_rocdbgapi.sh - Builds ROCm Debugger API.
build_gdb.sh - Builds ROC gdb.
build_roctracer.sh - Builds ROC gdb. UNDER DEVELOPMENT
create_release_tarball.sh - This builds an important release artifact
containing all sources.
These scripts install into $HOME/rocm/aomp (or $AOMP if set).
The clone_aomp.sh script clones the necessary repositories and the correct branches into subdirectories of $HOME/git/aomp11 (or $AOMP_REPOS if set) The repositories needed by AOMP are shown in the following table. The first column is the AOMP component that uses the repositories.| Component | SUBDIRECTORY | REPOSITORY LINKS |
|---|---|---|
| roct | $HOME/git/aomp11/roct-thunk-interfaces | roct-thunk-interfaces |
| rocr | $HOME/git/aomp11/rocr-runtime | rocr-runtime |
| llvm-project | $HOME/git/aomp11/amd-llvm-project | llvm-project |
| extras | $HOME/git/aomp11/aomp-extras | aomp-extras |
| vdi | $HOME/git/aomp11/vdi | vdi |
| comgr | $HOME/git/aomp11/rocm-compilersupport | comgr |
| hipvdi | $HOME/git/aomp11/hip-on-vdi | hipvdi |
| ocl | $HOME/git/aomp11/opencl-on-vdi | ocl |
| openmp | $HOME/git/aomp11/llvm-project/openmp | llvm-project/openmp |
| libdevice | $HOME/git/aomp11/rocm-device-libs | rocm-device-libs |
| flang | $HOME/git/aomp11/flang | flang |
| rocminfo | $HOME/git/aomp11/rocminfo | rocminfo |
| rocdbgapi | $HOME/git/aomp11/ROCdbgapi | rocdbgapi |
| rocgdb | $HOME/git/aomp11/ROCgdb | rocgdb |
| roctrace | $HOME/git/aomp11/roctracer | roctracer |
| rocprofiler | $HOME/git/aomp11/rocprofiler | rocprofiler |
The AOMP build scripts use many environment variables to control how AMOP is built. The file aomp_common_vars is sourced by all build scripts to provide consistent values and default values for all environment variables. There are generous comments in aomp_common_vars that provide information about the variables. Developers should take some time to read comments in the aomp_common_vars file.
These are some important environment variables and their default values.
AOMP $HOME/rocm/aomp
CUDA /usr/local/cuda
AOMP_REPOS $HOME/git/aomp11
AOMP_STANDALONE_BUILD 1
AOMP_VERSION 11.11
NVPTXGPUS 30,35,50,60,61,70
GFXLIST gfx700 gfx701 gfx801 gfx803 gfx900 gfx902 gfx906 gfx908
BUILD_TYPE Release
You can override any environment variable by setting values in your .bashrc or .bash_profile. Here is a sample of commands you might want to put into your .bash_profile:
AOMP=/tmp/$USER/git/aomp
BUILD_TYPE=Debug
NVPTXGPUS=30,35,70
GFXLIST="gfx803 gfx906"
AOMP_VERSION="12.0"
export AOMP BUILD_TYPE NVPTXGPUS GFXLIST
To build and clone all components using the latest development sources, first clone aomp repo and checkout the amd-stg-openmp branch as follows:
mkdir $HOME/git
cd git
git clone https://github.com/ROCm-Developer-Tools/aomp.git aomp11
cd aomp11
git checkout amd-stg-openmp
./clone_aomp.sh
The first time you run ./clone_aomp.sh, it could take a long time to clone the repositories. Subsequent calls will only pull the latest updates.
WARNING: The script clone_aomp.sh does not pull updates for this aomp repository. You must pull aomp repository manually. So please run this frequently to stay current with the aomp development team.
cd $HOME/git/aomp11/aomp
git pull
./clone_aomp.sh
Building from source requires many prerequisites. Follow all the instructions here to satisfy the many build-from-source prerequisites. If you are on a system that another user has verified all the build-from-source requirements, you still must install individual python components identified in section "2. User-installed Python Components".
After you have all the source repositories from running clone_aomp.sh and you have all the prerequisites installed, run the master build script that steps through the build of all aomp components:
./build_aomp.sh
The default AOMP source build is a standalone build of all components needed for compilation and execution with the exception of the kfd Linux kernel module for AMD GPUs or the CUDA SDK for Nvidia GPUs.
Starting with ROCM 3.0 there is a version of AOMP distributed with ROCm with the package name aomp-amdgpu. The ROCm aomp-amdgpu package installs into /opt/rocm/aomp with the command 'apt-get install aomp-amdgpu'. Unlike the AOMP standalone build, ROCm aomp-amdgpu is integrated into ROCm. It depends and uses existing ROCm components. It was build with AOMP_STANDALONE_BUILD=0.
Eventually, the ROCm production compiler will have complete OpenMP support and the ROCm aomp-amdgpu package will be deprecated. That ROCm production compiler has the package name llvm-amdgp and installs in /opt/rocm/llvm.
Starting with ROCM 4.0, AOMP will continue as a research and development compiler released into github with the package name aomp. This will often be used to validate code going into the ROCm production compiler including quick fixes for issues identified in [AOMP issues] (https://github.com/ROCm-Developer-Tools/aomp/issues). To ensure complete isolation from the ROCm installation and to make AOMP work in the absense of ROCm, all necessary ROCm components are built from source. Furthermore, through comprehensive use of RPATH by AOMP, all AOMP libraries and references to AOMP libraries will resolve the absolute location within the AOMP installation. AOMP never requires the use of LD_LIBRARY_PATH. It is reserved for user library development. This strategy prevents any interference with a ROCm installation. Libraries that may have been installed by a ROCm installation including roct and rocr, will not be used by AOMP. Nor will the AOMP installation affect any ROCm component.
The only ROCm common component required by AOMP is the kernel kfd.
To build aomp, run the master build script build_aomp.sh or run these individual scripts in this order:
# Start with these components if building AOMP standalone
# AOMP_STANDALONE_BUILD==1 is the default.
# If you turn this off, you must have a compatible ROCm installed.
./build_roct.sh
./build_roct.sh install
./build_rocr.sh
./build_rocr.sh install
# These 6 core components are always built and depend on a
# ROCm installation. These are the only components built with
# AOMP_STANDALONE_BUILD==0.
./build_project.sh
./build_project.sh install
./build_extras.sh
./build_extras.sh install
./build_openmp.sh
./build_openmp.sh install
./build_pgmath.sh
./build_pgmath.sh install
./build_flang.sh
./build_flang.sh install
./build_flang_runtime.sh
./build_flang_runtime.sh install
./build_libdevice.sh
./build_libdevice.sh install
./build_comgr.sh
./build_comgr.sh install
./build_rocminfo.sh
./build_rocminfo.sh install
# These only build on x86_64
./build_vdi.sh
./build_vdi.sh install
./build_hipvdi.sh
./build_hipvdi.sh install
./build_ocl.sh
./build_ocl.sh install
# These two components build only when
# AOMP_BUILD_STANDALONE==1 && AOMP_BUILD_DEBUG==1
# which are the default values.
./build_rocdbgapi.sh
./build_rocdbgapi.sh install
./build_rocgdb.sh
./build_rocgdb.sh install
Once you have a successful development build, individual components can be incrementally rebuilt without rebuilding the entire system or the entire component. For example, if you change a file in the llvm-project repository. Run this command to incrementally build llvm, clang, and lld and update your installation.
./build_project.sh install
WARNING: When the build scripts are run with NO arguments (that is, you do not specify "install" or "nocmake"), the build scripts will rebuild the entire component by DELETING THE BUILD DIRECTORY before running cmake and make. This is called a fresh start.
The build scripts will build from the source directories identified by the environment variable AOMP_REPOS. The AOMP_REPOS default value is $HOME/git/aomp11. The out-of-source build directory for each component is $AOMP_REPOS/build/<component_name>.
The install location is defined by the $AOMP environment variable. The value of AOMP MUST be reserved as a symbolic link. That is, the physical installation will be in directory name formed by concatonating the version string to the value of $AOMP. The "build_project.sh install" script will make a symbolic link from the physical directory to the symbolic directory $AOMP. The default value for AOMP is $HOME/rocm/aomp. For example, when building AOMP version 11.9-0 the install scripts will put all files and directories in $HOME/rocm/aomp_11.9-0 and create a symbolic link as follows:
ln -sf ${AOMP}_11.9-0 ${AOMP}
All testing for AOMP uses the environment variable AOMP to locate the installation. This makes it easy to switch between versions of AOMP for testing by simply changing the environment variable AOMP. You do NOT need to change the symbolic link. For example, if the aomp symbolic link currently points to aomp_11.11-2 and you want to test aomp_11.8-0, do this:
export AOMP=$HOME/rocm/aomp_11.8-0
The aomp package installs in /usr/lib/aomp_<version_string> and symlinks /usr/lib/aomp to this directory. To test the installed package, you can set AOMP to /usr/lib/aomp or /usr/lib/aomp_<version_string>.
It is also possible to test the ROCm compiler with AOMP tests by setting AOMP as follows:
export AOMP=/opt/rocm/llvm
Many tests must determine the type of GPU available with the mygpu script. The location of mygpu is expected in $AOMP/bin/mygpu. ROCm puts this utility in a different location. But this script is only used if the environment variable AOMP_GPU is not set which is the default. So, to use AOMP tests with the ROCM compiler set the value of AOMP_GPU as follows:
export AOMP_GPU=`/opt/rocm/bin/mygpu`
Of the many repos identified above, AOMP developers may only change amd-llvm-project, aomp, aomp-extras, and flang. AOMP required changes to other non-AOMP components must be patched. The patches/README.md escribes the AOMP patch mechanism in detail.