-
Notifications
You must be signed in to change notification settings - Fork 26
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Integrating AIR with the ROCm runtime #367
Changes from 6 commits
1f4668f
dbe046c
db6b895
6ba1136
ef3d3be
7e2b322
04a27d4
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -27,6 +27,27 @@ jobs: | |
fetch-depth: 2 | ||
submodules: "true" | ||
|
||
- name: Install necessary build tools for libelf | ||
run: sudo apt install autoconf flex bison gawk autopoint -y | ||
|
||
- name: Install necessary build tools for the ROCm runtime | ||
run: | | ||
sudo apt install libelf-dev elfutils libdwarf-dev | ||
sudo mkdir --parents --mode=0755 /etc/apt/keyrings | ||
wget https://repo.radeon.com/rocm/rocm.gpg.key -O - | \ | ||
gpg --dearmor | sudo tee /etc/apt/keyrings/rocm.gpg > /dev/null | ||
for ver in 5.3.3 5.4.3 5.5.1 5.6; do | ||
echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/rocm/apt/$ver focal main" \ | ||
| sudo tee --append /etc/apt/sources.list.d/rocm.list | ||
done | ||
echo -e 'Package: *\nPin: release o=repo.radeon.com\nPin-Priority: 600' \ | ||
| sudo tee /etc/apt/preferences.d/rocm-pin-600 | ||
sudo apt update | ||
sudo apt install rocm-hip-runtime-dev5.6.0 | ||
|
||
- name: Install libboost | ||
run: sudo apt-get install -y libboost-all-dev | ||
|
||
- name: Install pip packages | ||
run: sudo pip install psutil pybind11 numpy | ||
|
||
|
@@ -66,12 +87,26 @@ jobs: | |
run: utils/clone-mlir-aie.sh | ||
shell: bash | ||
|
||
- name: Get ROCm-air-platforms | ||
id: clone-rocm-air-platforms | ||
run: utils/clone-rocm-air-platforms.sh | ||
shell: bash | ||
|
||
- name: Rebuild and Install libxaie | ||
run: utils/github-clone-build-libxaie.sh | ||
|
||
- name: Rebuild and Install elfutils | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Also part of the libelf change |
||
run: utils/github-clone-build-elfutils.sh | ||
|
||
- name: Rebuild and Install mlir-aie | ||
run: utils/github-build-mlir-aie.sh | ||
|
||
- name: Build ROCt | ||
run: ROCm-air-platforms/utils/clone-build-roct.sh | ||
|
||
- name: Build ROCr | ||
run: ROCm-air-platforms/utils/clone-build-rocr.sh | ||
|
||
# Build the repo test target in debug mode to build and test. | ||
- name: Build and test (Assert) | ||
run: | | ||
|
@@ -86,7 +121,10 @@ jobs: | |
-DMLIR_DIR=../llvm/install/lib/cmake/mlir/ \ | ||
-DLLVM_DIR=../llvm/install/lib/cmake/llvm/ \ | ||
-DAIE_DIR=`pwd`/../mlir-aie/install/lib/cmake/aie/ \ | ||
-DELFUTILS_DIR=$PWD/../elfutils \ | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Also part of the libelf change |
||
-DLibXAIE_ROOT=`pwd`/../aienginev2/install \ | ||
-Dhsa-runtime64_DIR=`pwd`/../rocm/lib/cmake/hsa-runtime64/ \ | ||
-Dhsakmt_DIR=`pwd`/../rocm/lib/cmake/hsakmt/ \ | ||
-DAIR_RUNTIME_TARGETS:STRING="x86_64" \ | ||
-Dx86_64_TOOLCHAIN_FILE=`pwd`/../cmake/modules/toolchain_x86_64.cmake \ | ||
-DLLVM_EXTERNAL_LIT=`pwd`/../llvm/build/bin/llvm-lit \ | ||
|
@@ -113,7 +151,10 @@ jobs: | |
-DMLIR_DIR=../llvm/install/lib/cmake/mlir/ \ | ||
-DLLVM_DIR=../llvm/install/lib/cmake/llvm/ \ | ||
-DAIE_DIR=`pwd`/../mlir-aie/install/lib/cmake/aie/ \ | ||
-DELFUTILS_DIR=$PWD/../elfutils \ | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Also part of the libelf change |
||
-DLibXAIE_ROOT=`pwd`/../aienginev2/install \ | ||
-Dhsa-runtime64_DIR=`pwd`/../rocm/lib/cmake/hsa-runtime64/ \ | ||
-Dhsakmt_DIR=`pwd`/../rocm/lib/cmake/hsakmt/ \ | ||
-DAIR_RUNTIME_TARGETS:STRING="x86_64" \ | ||
-Dx86_64_TOOLCHAIN_FILE=`pwd`/../cmake/modules/toolchain_x86_64.cmake \ | ||
-DLLVM_EXTERNAL_LIT=`pwd`/../llvm/build/bin/llvm-lit \ | ||
|
This file was deleted.
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -150,7 +150,10 @@ foreach(target ${AIR_RUNTIME_TARGETS}) | |
-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} | ||
-DCMAKE_INSTALL_PREFIX=${CMAKE_INSTALL_PREFIX} | ||
-DLibXAIE_ROOT=${LibXAIE_ROOT} | ||
-Dhsa-runtime64_DIR=${hsa-runtime64_DIR} | ||
-Dhsakmt_DIR=${hsakmt_DIR} | ||
-DAIE_DIR=${AIE_DIR} | ||
-DELFUTILS_DIR=${ELFUTILS_DIR} | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Also part of the libelf change |
||
BUILD_ALWAYS true | ||
STEP_TARGETS clean build install test | ||
USES_TERMINAL_CONFIGURE true | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,10 @@ | ||
# Building AIR Tools and Runtime | ||
|
||
## Running on the VCK5000 | ||
|
||
The first step required for running on the VCK5000 is building the hardware platform and loading it on your device. We provide a platform repository titled [ROCm-air-platforms](https://github.com/Xilinx/ROCm-air-platforms) which contains instructions on how to compile and load the platform, driver, and firmware on your VCK5000. Then, the AIR tools must be installed by following the [Building external projects on X86](#building-external-projects-on-X86) and [Environment setup](#environment-setup) instructions. Then, the [AIR PCIe driver](https://github.com/Xilinx/ROCm-air-platforms/tree/main/driver) from [ROCm-air-platforms](https://github.com/Xilinx/ROCm-air-platforms) must be compiled and installed. Now you are ready to run tests using AIR on the VCK5000! For example, go to [Test 13](test/13_mb_add_one) and run `make` followed by `sudo ./test.elf` to run the application. | ||
|
||
|
||
## Prerequisites | ||
|
||
**mlir-air** is built and tested on Ubuntu 20.04 with the following packages installed: | ||
|
@@ -21,7 +26,7 @@ In addition the following packages maybe useful: | |
Xilinx aienginev2 library from https://github.com/Xilinx/embeddedsw | ||
|
||
``` | ||
Vivado and Vitis are used to build the platforms for available Xilinx cards with AIEs (VCK190 and VCK5000). Vitis also contains the AIE single core compiler xchesscc. The aienginev2 library is a driver backend used to configure the AIE array and used in building the AIR runtime. The aietools single core compiler and aienginev2 library are required to build binaries that run on AIE hardware. | ||
Vivado and Vitis are used to build the platforms for available Xilinx cards with AIEs (VCK5000). Vitis also contains the AIE single core compiler xchesscc. The aienginev2 library is a driver backend used to configure the AIE array and used in building the AIR runtime. The aietools single core compiler and aienginev2 library are required to build binaries that run on AIE hardware. | ||
|
||
NOTE: using the Vitis recommended settings64.sh script to set up your environement can cause tool conflicts. Setup your environment in the following order for aietools and Vitis: | ||
|
||
|
@@ -34,7 +39,20 @@ NOTE: that you do not need the above additional Xilinx packages to make use of t | |
Building mlir-air requires several other open source packages: | ||
- [mlir](https://github.com/llvm/llvm-project/tree/main/mlir) | ||
- [mlir-aie](https://github.com/Xilinx/mlir-aie) | ||
- [Xilinx cmakeModules](https://github.com/Xilinx/cmakeModules) | ||
- [Xilinx cmakeModules](https://github.com/Xilinx/cmakeModules). Note that this is installed as a submodule of mlir-aie. | ||
|
||
In order to build and run on PCIe cards, you first have to build and install the aienginev2 library. We chose to install the library in /opt/xaiengine but it is not required for the tools to be installed there. Just ensure that when building mlir-aie and mlir-air, that you point to the directory in which the aienginev2 library was installed. | ||
|
||
``` | ||
git clone https://github.com/jnider/aie-rt | ||
cd aie-rt | ||
git checkout joel-aie | ||
cd driver/src | ||
make -f Makefile.Linux CFLAGS="-D__AIELINUX__ -D__AIESYSFS__ -D__AIEAMDAIR__" | ||
sudo cp -r ../include /opt/aiengine/ | ||
sudo cp libxaiengine.so* /opt/xaiengine/lib/ | ||
export LD_LIBRARY_PATH=/opt/xaiengine/lib:${LD_LIBRARY_PATH} | ||
``` | ||
|
||
## Building external projects on X86 | ||
|
||
|
@@ -46,127 +64,51 @@ First, run utils/setup_python_packages.sh to setup the prerequisite python packa | |
source utils/setup_python_packages.sh | ||
``` | ||
|
||
Next, clone and build LLVM, with the ability to target AArch64 as a cross-compiler, and with MLIR enabled. In addition, we make some common build optimizations to use a linker ('lld' or 'gold') other than 'ld' (which tends to be quite slow on large link jobs) and to link against libLLVM.so and libClang so. You may find that other options are also useful. Note that due to changing MLIR APIs, only a particular revision is expected to work. | ||
Next, clone and build LLVM, with MLIR enabled. In addition, we make some common build optimizations to use a linker ('lld' or 'gold') other than 'ld' (which tends to be quite slow on large link jobs) and to link against libLLVM.so and libClang so. You may find that other options are also useful. Note that due to changing MLIR APIs, only a particular revision is expected to work. | ||
|
||
To clone llvm and cmakeModules, see utils/clone-llvm.sh for the correct commithash. We point LLVM and subsequent tools to a common installation directory. | ||
Run the following to clone and build llvm: | ||
|
||
``` | ||
cd utils | ||
./clone-llvm.sh | ||
./build-llvm-local.sh llvm build ../../install | ||
cd mlir-air | ||
./utils/clone-llvm.sh | ||
./utils/build-llvm-local.sh llvm | ||
``` | ||
|
||
Next, clone and build MLIR-AIE with paths to llvm, aienginev2, and cmakeModules repositories. Again, we use a common installation directory. Note that in the following commands, we assume that the aienginev2 library is installed in /opt/xaiengine as directed in the `Building on x86 with runtime for PCIe` section. If the aienginev2 library was installed elsewhere, be sure that the 4th argument to build mlir-aie points to that location. | ||
Next, we are going to need to clone and build elfutils. This is used in the runtime to unpack and decode AIE application binaries. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Also part of the libelf change |
||
``` | ||
./clone-mlir-aie.sh | ||
./build-mlir-aie-local.sh llvm mlir-aie/cmake/modulesXilinx /opt/xaiengine mlir-aie build ../../install | ||
./utils/github-clone-build-elfutils.sh | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Also part of the libelf change |
||
``` | ||
|
||
The MLIR-AIE tools will be able to generate binaries targeting AIEngines. | ||
|
||
Finally, build the MLIR-AIR tools for your desired use case: | ||
|
||
- Building on x86 | ||
- Building on x86 for deployment on a PCIe platform with AIEs | ||
- Building on x86 for deployment on an embedded platform with AIEs | ||
- Building on ARM for deployment on an embedded platform with AIEs | ||
|
||
## Building on x86 | ||
|
||
Use the following command to build the AIR tools to compile on x86: | ||
Next, clone and build MLIR-AIE with paths to llvm, aienginev2, and cmakeModules repositories. Note that in the following commands, we assume that the aienginev2 library is installed in /opt/xaiengine as directed in the `Prerequisites` section. If the aienginev2 library was installed elsewhere, be sure that the 4th argument to build mlir-aie points to that location. | ||
|
||
``` | ||
./build-mlir-air.sh $SYSROOT /full/path/to/mlir-air/utils/llvm /full/path/to/mlir-air/utils/cmakeModules /full/path/to/mlir-air/utils/mlir-aie build install | ||
./utils/clone-mlir-aie.sh | ||
./utils/build-mlir-aie-local.sh llvm mlir-aie/cmake/modulesXilinx /opt/xaiengine mlir-aie | ||
``` | ||
|
||
## Building on x86 with runtime for PCIe | ||
|
||
In order to build and run on PCIe cards, you first have to build and install the aienginev2 library. We chose to install the library in /opt/xaiengine but it is not required for the tools to be installed there. Just ensure that when building mlir-aie and mlir-air, that you point to the directory in which the aienginev2 library was installed. | ||
Use the following command to build the AIR tools to compile on x86 for PCIe cards (VCK5000). Make sure that ${ROCM\_ROOT} is pointing to your local ROCm install: | ||
|
||
``` | ||
git clone https://github.com/jnider/aie-rt | ||
cd aie-rt | ||
git checkout joel-aie | ||
cd driver/src | ||
make -f Makefile.Linux CFLAGS="-D__AIELINUX__ -D__AIESYSFS__ -D__AIEAMDAIR__" | ||
sudo cp -r ../include /opt/xaiengine/ | ||
sudo cp libxaiengine.so* /opt/xaiengine/lib/ | ||
export LD_LIBRARY_PATH=/opt/xaiengine/lib:${LD_LIBRARY_PATH} | ||
./utils/build-mlir-air-pcie.sh llvm/ mlir-aie/cmake/modulesXilinx/ mlir-aie/ /opt/xaiengine elfutils/ ${ROCM_ROOT}/lib/cmake/hsa-runtime64/ ${ROCM_ROOT}/lib/cmake/hsakmt/ | ||
``` | ||
|
||
Use the following command to build the AIR tools to compile on x86 for PCIe cards (VCK5000): | ||
The PCIe AIR runtime requires the use of the [AIR PCIe kernel driver](https://github.com/Xilinx/ROCm-air-platforms/tree/main/driver). The driver directory in the [ROCm-air-platforms](https://github.com/Xilinx/ROCm-air-platforms) repository contains documentation on how to compile and load the AIR PCIe kernel driver. Run the following script to clone the ROCm-air-platforms repository: | ||
|
||
``` | ||
./utils/build-mlir-air-pcie.sh utils/llvm/ utils/mlir-aie/cmake/modulesXilinx/ utils/mlir-aie/ /opt/xaiengine | ||
./utils/clone-rocm-air-platforms.sh | ||
``` | ||
|
||
The PCIe AIR runtime requires the use of the [AIR PCIe kernel driver](driver). The driver directory contains documentation on how to compile and load the AIR PCIe kernel driver. | ||
|
||
Note that building a sysroot and the cross-compilation instructions are not necessary when running on x86 with the PCIe runtime. | ||
|
||
## Environment setup | ||
|
||
Set up your environment to use the tools you just built with the following commands: | ||
|
||
``` | ||
export PATH=/path/to/mlir-air/install/bin:${PATH} | ||
export PYTHONPATH=/path/to/mlir-air/install/python:${PYTHONPATH} | ||
export LD_LIBRARY_PATH=/path/to/mlir-air/install/lib:/opt/xaiengine/lib:${LD_LIBRARY_PATH} | ||
``` | ||
|
||
Note that if you are running on x86 with the PCIe runtime, the following path should be added to your path rather than `/path/to/mlir-air/install/bin`. | ||
``` | ||
export LD_LIBRARY_PATH=/path/to/install/mlir-air/lib:/opt/xaiengine/lib:${LD_LIBRARY_PATH} | ||
export PATH=/path/to/mlir-air/install-pcie/bin:${PATH} | ||
``` | ||
|
||
## Building hardware platforms | ||
|
||
The instructions for building the hardware platform designs are found in the mlir-air/platforms directory: | ||
|
||
- [xilinx_vck190_air](platforms/xilinx_vck190_air) | ||
- [xilinx_vck5000_air](platforms/xilinx_vck5000_air) | ||
- [xilinx_vck5000_air_scale_out](platforms/xilinx_vck5000_air_scale_out) | ||
|
||
## Building a Sysroot | ||
|
||
Since the MLIR-AIE tools are cross-compiling, in order to actually compile code, we need a 'sysroot' directory, | ||
containing an ARM rootfs. This rootfs must match what will be available in the runtime environment. | ||
Note that copying the rootfs is often insufficient, since many root file systems include absolute links. | ||
Absolute symbolic links can be converted to relative symbolic links using [symlinks](https://github.com/brandt/symlinks). | ||
|
||
```sh | ||
cd / | ||
sudo symlinks -rc . | ||
``` | ||
Following the VCK190 platform build steps will also create a sysroot. | ||
|
||
## Compiling Runtime for AArch64 (partial cross-compile) | ||
|
||
Use the following command to build the AIR tools to build on x86 and cross-compile host code for ARM: | ||
|
||
``` | ||
??? | ||
``` | ||
|
||
## Compiling Tools and Runtime for AArch64 (full cross-compile) | ||
|
||
Use the following command to cross-compile the AIR tools for ARM: | ||
|
||
``` | ||
cd utils | ||
./cross-build-llvm.sh /full/path/to/mlir-air/install-aarch64 | ||
./cross-build-mlir-aie.sh ../cmake/modules/toolchain_crosscomp_aarch64.cmake $SYSROOT /full/path/to/mlir-air/utils/cmakeModules /full/path/to/mlir-air/utils/llvm /full/path/to/mlir-air/install-aarch64 | ||
./cross-build-mlir-air.sh ../cmake/modules/toolchain_crosscomp_aarch64.cmake $SYSROOT /full/path/to/mlir-air/utils/cmakeModules /full/path/to/mlir-air/utils/llvm /full/path/to/mlir-air/utils/mlir-aie /full/path/to/mlir-air/install-aarch64 | ||
cd .. | ||
tar -cvf air_tools.tar.gz install-aarch64 | ||
``` | ||
|
||
## Running on the VCK5000 | ||
|
||
The first step required for running on the VCK5000 is building the hardware platform and loading it on your device. [Here](platforms/xilinx_vck5000_air) we provide documentation on how to do that. Then, the AIR tools must be installed by running the [Building external projects on X86](#building-external-projects-on-X86) followed by the [Building on x86 with runtime for PCIe](#building-on-x86-with-runtime-for-PCIe) and [Environment setup](#environment-setup). Then, the [AIR PCIe driver](driver) must be compiled and installed. Now you are ready to run tests using AIR on the VCK5000! For example, go to [Test 13](test/13_mb_add_one) and run `make` followed by `sudo ./test.elf` to run the application. | ||
|
||
|
||
|
||
----- | ||
|
||
<p align="center">Copyright© 2022 Advanced Micro Devices, Inc.</p> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We recently decided to replace our custom libelf with the pre-built one. It means these tools are no longer necessary. I don't know if it's better to make that change in a different commit or to edit this one. What do you think?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can edit the commit. Thanks for the comments I see how you are doing it in your mlir-aie fork.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In the moment felt better as a new commit so put in 04a27d4. Let me know what you think. I couldn't get CMake to automatically find LibElf if you know how to do that that could make it cleaner, but I am just pointing to the standard install location which also works.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't know how to get CMake to search for a library. We could pass the path explicitly (like I was doing for the custom version) but with a default to the standard Ubuntu location. I think for now, relying on the standard Ubuntu location (as you are doing) is acceptable. If everyone will just use 'apt install' then it is highly unlikely the library will be at a different location.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
100% agree. Sweet will merge this in. Thanks!