AOMP Release 19.0-2

These are the release notes for AOMP 19.0-2 AOMP uses AMD developer modifications to the upstream LLVM development trunk. These differences are managed in a branch called the "amd-staging". This branch is found in a mirror of upstream LLVM found at https://github.com/ROCm/llvm-project. The amd-staging branch is constantly changing as it merges the upstream development trunk with its downstream development updates. The AMD modifications are experimental and/or/while contributions under review for the upstream trunk. AOMP uses a snapshot of amd-staging at the commit ids and dates listed below. AOMP also includes builds of related ROCm components. We call AOMP a "standalone" build as it does not use or require ROCm with the exception of the kernel module (dkms) and libdrm which are often part of the Linux distribution. AOMP is isolated from any ROCm installations by installing into /usr/lib/aomp and the use of RPATH for runtime libraries.

For AOMP 19.0-2, the last trunk commit is 6012de2b4ec24826574fe9f2d74c7d2ff2b52f23on June 20, 2024. The last amd-only commit is c3a455408b118b8c22f23c7a65d2b5dbf491ab56 on June 20, 2024. These commits forms a frozen branch now called "aomp-19.0-2". See https://github.com/ROCm/llvm-project/tree/aomp-19.0-2.

The integrated ROCm components for this AOMP release were built with ROCM 6.1.2 sources.
This is the 2nd AOMP release based on LLVM 19 development.

AOMP 19.0-1 was tagged, but will not be released.

Changes since AOMP 19.0-0:

• Significant runtime features to support zero-copy for CPU-GPU unified shared memory. See subsections below.
• Merge of the LLVM upstream relocation of libomptarget into the high level "offload" directory. This establishes the long term objective of the LLVM community to unify offload support for different offloading programming models.
• The integrated ROCm components (non-compiler) were built from ROCM 6.1.2 sources.
• Significant enhancements to the gpurun utility including: support for multiple devices, heterogeneous devices, malloc control inherited from numa-ctl -m -l options, and CPU core binding to same numa node as selected GPU. These changes preserve gpurun's ability to oversubscribe (run multiple processes per GPU) by segmenting a GPUs CUs to different processes. To be fixed in 19.0-3, gpurun fails in TPX mode on MI300X.
• Changes in runtime library locations unique to CPU target triple including fixes for lib64 in Red Hat package.
• Support for fp16 and bfloat16 reductions
• Removed long deprecated utilities mygpu, mymcpu, aompcc, aompExtractRegion, clang-ocl, and cloc.sh.

Errata for AOMP 19.0-2

• gpurun fails in TPX mode for MI300X
• LIBOMPTARGET_SYNC_COPY_BACK default is still true. This is to circumvent a long-standing SDMA problem where signal values appear incorrect to SDMA engines.
• Failure in dynamic_module_load which impacts application termination that uses offloading in shared objects.

Implicit Zero-Copy behavior on MI300A

OpenMP provides a relaxed shared memory model. Map clauses provided in the source code indicate how data is used and copied to and from the GPU device for each target region. On GPUs that provide unified shared memory like the MI300A, these clauses are optional but provide portability to discreet memory GPUs. There is an OpenMP pragma called "requires unified_shared_memory" which tell the compiler and runtime that the code is NOT portable to discreet memory GPUs, and must be compiled and executed on GPUs such as the MI300A. The MI300A is one of several AMD GPUs that has a feature to disable/enable page migration between CPU and GPU called xnack. In this release of the compiler and runtime, we set the runtime behavior depending on the status of xnack and existence of the pragma "requires unified_shared_memory".

MI300A	NO requires unified_shared_memory	requires unified_shared_memory
XNACK enabled	Implicit Zero-Copy	Zero-Copy
XNACK disabled	Copy	Runtime warning*

(*) The runtime warning when running an application using #pragma omp requires unified_shared_memory in XNACK disabled mode can be turned into a runtime error by setting environment variable OMPX_STRICT_SANITY_CHECKS to true (e.g., OMPX_STRICT_SANITY_CHECKS=true ./app_exec).

Implicit Zero-Copy on MI200 and MI300X and any other discrete GPU:

• On discrete memory GPUs, for applications not using #pragma omp requires unified_shared_memory, turn on implicit zero-copy behavior by running applications in XNACK enabled environment and setting to true the environment variable OMPX_APU_MAPS (e.g. HSA_XNACK=1 OMPX_APU_MAPS=1 ./app_exec)
• All other configurations, for applications not using #pragma omp requires unified_shared_memory, will be run in copy behavior.

MI200, MI300X, etc.	not unified_shared_memory	unified_shared_memory
XNACK enabled and OMPX_APU_MAPS=1	Implicit Zero-Copy	Zero-Copy
XNACK enabled	Copy	Zero-Copy
XNACK disabled	Copy	Runtime warning(*)

MI300A host memory pre-faulting in Zero-Copy modes

On MI300A, host memory TLB prefaulting applies when running in in Implicit Zero-Copy and when using #pragma omp requires unified_shared_memory

• By default, for all memory copies with size larger or equal to 1MB, the OpenMP runtime makes the copied host memory visible to the target device agent before calling the copy function
• The environment variable LIBOMPTARGET_APU_PREFAULT_MEMCOPY controls this behavior and it is set to true by default. Setting it to false will disable prefaulting for all memory copy sizes (e.g., disable prefaulting with LIBOMPTARGET_APU_PREFAULT_MEMCOPY=false ./app_exec)
• The environment variable LIBOMPTARGET_APU_PREFAULT_MEMCOPY_SIZE controls the minimum size after which prefaulting is performed. It is currently set to 1MB, meaning that all memory copies that are performed in a synchronous way will have the host memory first prefaulted. Changing the minimum size enables prefaulting at sizes different than larger or equal to 1MB (e.g., to prefault all memory copies larger than 1KB, run with LIBOMPTARGET_APU_PREFAULT_MEMCOPY_SIZE=1024 ./app_exe)

rocm-6.1.2

ROCm release v6.1.2

rocm-6.1.1

ROCm release v6.1.1

rocm-6.1.0

ROCm release v6.1.0

AOMP Release 19.0-0

These are the release notes for AOMP 19.0-0. AOMP uses AMD developer modifications to the upstream LLVM development trunk. These differences are managed in a branch called the "amd-staging". This branch is found in a mirror of upstream LLVM found at https://github.com/ROCm/llvm-project. The amd-staging branch is constantly changing as it merges the upstream development trunk with its downstream development updates. The AMD modifications are experimental and/or/while contributions under review for the upstream trunk. AOMP uses a snapshot of amd-staging at the commit ids and dates listed below. AOMP also includes builds of related ROCm components. We call AOMP a "standalone" build as it does not use or require ROCm with the exception of the kernel module (dkms) and libdrm which are often part of the Linux distribution. AOMP is isolated from any ROCm installations by installing into /usr/lib/aomp and the use of RPATH for runtime libraries.

For AOMP 19.0-0, the last trunk commit is 601e102bdb55e12a2f791e0d68fd6f81ffc21e21 on March 17, 2024. The last amd-only commit is a3c2cd57a6f99709d61d35d17527c84d1af0c780 on March 16, 2024. These commits forms a frozen branch now called "aomp-19.0-0". See https://github.com/ROCm/llvm-project/tree/aomp-19.0-0.

The integrated ROCm components for this AOMP release were built with ROCM 6.0.2 sources.
This is the 1st AOMP release based on LLVM 19 development.

These are the changes since AOMP 18.0-1:

• Now use ROCM 6.0.2 sources for non compiler components.
• Default to not use multiple SDMA engines. This could be changed for testing by setting:
  LIBOMPTARGET_AMDGPU_USE_MULTIPLE_SDMA_ENGINES=true
• Updates to gpurun utility
  • uses preset HSA_CU_MASK and ROCR_VISIBLE_DEVICES.
• OpenMP lit testing pre-install, 100% pass rate.
• [Perf] Implement the loop directive as used in the metadirective for spec.
• Support for testing QMCpack NiO performance.
• ROCgdb support for gfx1103.
• Circumvented driver timing issue with complex signal chains. Post kernel copies occasionally started before kernel completion. The temporary fix forces the wait for kernel completion before scheduling post-kernel copies. For testing purposes, this behavior could be reverted by setting LIBOMPTARGET_SYNC_COPY_BACK=false

Fix regressions:
#616 llvm-addr2line does not work on AOMP-generated binary.

Known Failures:

• Smoke
  • targetid_multi_image
  • get_mapped_ptr

rocm-6.0.2

ROCm release v6.0.2

AOMP Release 18.0-1

These are the release notes for AOMP 18.0-1. AOMP uses AMD developer modifications to the upstream LLVM development trunk. These differences are managed in a branch called the "amd-stg-open". This branch is found in a mirror of upstream LLVM found at https://github.com/RadeonOpenCompute/llvm-project. The amd-stg-open branch is constantly changing as AMD merges upstream development trunk with its internal open development efforts. The AMD modifications are experimental and/or/while contributions under review for the upstream trunk. AOMP uses a snapshot of amd-stg-open at the commit ids and dates listed below. AOMP also includes builds of related ROCm components. We call AOMP a "standalone" build as it does not use or require ROCm with the exception of the kernel module (dkms) and libdrm which are often part of the Linux distribution. AOMP is isolated from any ROCm installations by installing into /usr/lib/aomp and the use of RPATH for runtime libraries.

For AOMP 18.0-1, the last trunk commit is 5f71aa9270c3d680babfbc6e766773d113c2a79a on January 9, 2023. The last amd-only commit is 71a82c97d882bfece529db41130ef7aaf9696a6e on January 9, 2023 . These commits forms a frozen branch now called "aomp-18.0-1". See https://github.com/RadeonOpenCompute/llvm-project/tree/aomp-18.0-1.

The integrated ROCm components for this AOMP release were built with ROCM 6.0 sources.
This is the 2nd AOMP release based on LLVM 18 development.

These are the changes since AOMP 18.0-0:

• Move to ROCm 6.0 sources.

• Python 3.8 minimum required for building of ompd.

• Change the default optimization level to -O2 if user does not specify any -O option on the command line. Before 18.0-1, the default was -O0 if none was specified.

• In this release we are requesting users to install the amdgpu kernel driver found in the ROCm 6.0 amdgpu-dkms package. Nothing prevents the use of an older driver, but for support with current and new problems, we ask users to upgrade to this kernel driver. Run this command to test that your amdgpu kernel driver is on the correct version "modinfo -F version amdgpu". That command should return the string "6.3.6".

• Remove dependency on ROCm libraries for rocgdb. Reminder: A goal of aomp is to be isolated from ROCm stack with the exception of rocm-dkms which installs the correct kernel driver.

• • The following issues were resolved in this release:
    • #601 AOMP 17.0.3 crashes at compile time with conjunction in if clause

• While the source build of aomp, supports ASAN (address sanitizer), the release build does not due to complexities in packaging asan. Therefore, the aomp release does not support ASAN.

• This release has many build enhancements meant for developers that build aomp from source. These changes have little impact to user of AOMP.

  • The build scripts for this release now build openmp host and device libraries using LLVM_ENABLE_RUNTIMES=openmp with the COMPONENT build_project.sh. This was formerly done in the COMPONENT build_openmp.sh. This change allows lit testing of openmp. Extra libraries found in lib-debug and lib-perf directories are still build with the COMPONENT build_openmp.sh. ROCm-device-libs are also built as an external llvm project, which was formerly done by build_libdevice.sh.
  • Developers can now use ccache to speed rebuild process. The use of ccache is the default. To turn this off set AOMP_USE_CCACHE=0.
  • The build scripts have added a hidden directory in the installation called .aomp_component_stats showing the files added to the installation by each build COMPONENT. The current list of build COMPONENTs for AOMP are: prereq project roct rocr openmp extras comgr rocminfo flang-legacy pgmath flang flang_runtime hipcc hipamd rocdbgapi rocgdb roctracer rocprofiler
  • Changes to build_supp.sh which builds prequisite and supplemental components needed for the build and/or testing include: change to version 6.0 for of hsa-amd-aqlprofile, update wget location of silo tarball, change to version 6.0.x for rocmsmilib.
  • Build optimizations to reduce the size of the installation. Previously release binaries had llvm libraries statically linked in. We are no linking in the shared library build of llvm, and this reduces overall size by ~ 50%.
  • Changes to the build of libomptarget DeviceRTL to be code object version agnostic. These changes reduce the amount of files installed on to one per architecture for the opaque-linker + the fat archive for the linker wrapper as well as improve build time for the libraries.

• The following enhancements were made to the AOMP test infrastructure:

  • Support for openmp lit testing.
  • New testing of UMT.
  • babelstream updates to support USM.
  • babelstream updates to support FORTRAN babelstream.

rocm-6.0.0

ROCm release v6.0.0

rocm-5.7.1

ROCm release v5.7.1

rocm-5.7.0

ROCm release v5.7.0