Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[OpenMP][Offloading][AMDGPU] Exclude nogpulib in lit.cfg #76702

Closed
wants to merge 1 commit into from

Conversation

ergawy
Copy link
Member

@ergawy ergawy commented Jan 2, 2024

In order to run offloading tests for AMDGPUs, we should not use the
nogpulib flag added by lit.cfg. This is already done for Nvidia GPUs
and seems to have been overlooked for AMD.

This is a follow-up to #76355, please review only the last commit.

@llvmbot
Copy link
Collaborator

llvmbot commented Jan 2, 2024

@llvm/pr-subscribers-flang-openmp
@llvm/pr-subscribers-mlir-openmp

@llvm/pr-subscribers-mlir

Author: Kareem Ergawy (ergawy)

Changes

In order to run offloading tests for AMDGPUs, we should not use the
nogpulib flag added by lit.cfg. This is already done for Nvidia GPUs
and seems to have been overlooked for AMD.

This is a follow-up to #76355, please review only the last commit.


Full diff: https://github.com/llvm/llvm-project/pull/76702.diff

3 Files Affected:

  • (modified) mlir/lib/Conversion/OpenMPToLLVM/OpenMPToLLVM.cpp (+6-5)
  • (modified) openmp/libomptarget/test/lit.cfg (+1-1)
  • (added) openmp/libomptarget/test/offloading/fortran/target_update.f90 (+50)
diff --git a/mlir/lib/Conversion/OpenMPToLLVM/OpenMPToLLVM.cpp b/mlir/lib/Conversion/OpenMPToLLVM/OpenMPToLLVM.cpp
index cd1cfb3b7686d0..730858ffc67a71 100644
--- a/mlir/lib/Conversion/OpenMPToLLVM/OpenMPToLLVM.cpp
+++ b/mlir/lib/Conversion/OpenMPToLLVM/OpenMPToLLVM.cpp
@@ -239,11 +239,11 @@ void mlir::configureOpenMPToLLVMConversionLegality(
   target.addDynamicallyLegalOp<
       mlir::omp::AtomicReadOp, mlir::omp::AtomicWriteOp, mlir::omp::FlushOp,
       mlir::omp::ThreadprivateOp, mlir::omp::YieldOp, mlir::omp::EnterDataOp,
-      mlir::omp::ExitDataOp, mlir::omp::DataBoundsOp, mlir::omp::MapInfoOp>(
-      [&](Operation *op) {
-        return typeConverter.isLegal(op->getOperandTypes()) &&
-               typeConverter.isLegal(op->getResultTypes());
-      });
+      mlir::omp::ExitDataOp, mlir::omp::UpdateDataOp, mlir::omp::DataBoundsOp,
+      mlir::omp::MapInfoOp>([&](Operation *op) {
+    return typeConverter.isLegal(op->getOperandTypes()) &&
+           typeConverter.isLegal(op->getResultTypes());
+  });
   target.addDynamicallyLegalOp<mlir::omp::ReductionOp>([&](Operation *op) {
     return typeConverter.isLegal(op->getOperandTypes());
   });
@@ -282,6 +282,7 @@ void mlir::populateOpenMPToLLVMConversionPatterns(LLVMTypeConverter &converter,
       RegionLessOpConversion<omp::YieldOp>,
       RegionLessOpConversion<omp::EnterDataOp>,
       RegionLessOpConversion<omp::ExitDataOp>,
+      RegionLessOpConversion<omp::UpdateDataOp>,
       RegionLessOpWithVarOperandsConversion<omp::DataBoundsOp>>(converter);
 }
 
diff --git a/openmp/libomptarget/test/lit.cfg b/openmp/libomptarget/test/lit.cfg
index 19c5e5c4572227..dca922dcfc6bce 100644
--- a/openmp/libomptarget/test/lit.cfg
+++ b/openmp/libomptarget/test/lit.cfg
@@ -132,7 +132,7 @@ elif config.operating_system == 'Darwin':
     config.test_flags += " -Wl,-rpath," + config.library_dir
     config.test_flags += " -Wl,-rpath," + config.omp_host_rtl_directory
 else: # Unices
-    if config.libomptarget_current_target != "nvptx64-nvidia-cuda":
+    if config.libomptarget_current_target != "nvptx64-nvidia-cuda" and config.libomptarget_current_target != "amdgcn-amd-amdhsa":
         config.test_flags += " -nogpulib"
     config.test_flags += " -Wl,-rpath," + config.library_dir
     config.test_flags += " -Wl,-rpath," + config.omp_host_rtl_directory
diff --git a/openmp/libomptarget/test/offloading/fortran/target_update.f90 b/openmp/libomptarget/test/offloading/fortran/target_update.f90
new file mode 100644
index 00000000000000..fb35c5a36ab0e5
--- /dev/null
+++ b/openmp/libomptarget/test/offloading/fortran/target_update.f90
@@ -0,0 +1,50 @@
+! Offloading test for the `target update` directive.
+
+! REQUIRES: flang, amdgcn-amd-amdhsa
+
+! RUN: %libomptarget-compile-fortran-run-and-check-generic
+program target_update
+    implicit none
+    integer :: x(1)
+    integer :: host_id
+    integer :: device_id(1)
+
+    INTERFACE
+        FUNCTION omp_get_device_num() BIND(C)
+            USE, INTRINSIC :: iso_c_binding, ONLY: C_INT
+            integer :: omp_get_device_num
+        END FUNCTION omp_get_device_num
+    END INTERFACE
+
+    x(1) = 5
+    host_id = omp_get_device_num()
+
+!$omp target enter data map(to:x, device_id)
+!$omp target
+    x(1) = 42
+!$omp end target
+
+    ! Test that without a `target update` directive, the target update to x is
+    ! not yet seen by the host.
+    ! CHECK: After first target regions and before target update: x = 5
+    print *, "After first target regions and before target update: x =", x(1)
+
+!$omp target
+    x(1) = 84
+    device_id(1) = omp_get_device_num()
+!$omp end target
+!$omp target update from(x, device_id)
+
+    ! Test that after the `target update`, the host can see the new x value.
+    ! CHECK: After second target regions and target update: x = 84
+    print *, "After second target regions and target update: x =", x(1)
+
+    ! Make sure that offloading to the device actually happened. This way we
+    ! verify that we didn't take the fallback host execution path.
+    ! CHECK: Offloading succeeded!
+    if (host_id /= device_id(1)) then
+        print *, "Offloading succeeded!"
+    else
+        print *, "Offloading failed!"
+    end if
+end program target_update

In order to run offloading tests for AMDGPUs, we should not use the
`nogpulib` flag added by `lit.cfg`. This is already done for Nvidia GPUs
and seems to have been overlooked for AMD.
@ergawy ergawy changed the title [flang][OpenMP][Offloading][AMDGPU] Exclude nogpulib in lit.cfg [OpenMP][Offloading][AMDGPU] Exclude nogpulib in lit.cfg Jan 2, 2024
@jhuber6
Copy link
Contributor

jhuber6 commented Jan 2, 2024

This was not an oversight, Nvidia (non-LTO version) was excluded because it doesn't use the static library version of the runtime. What failures is this causing? The initial concern was that without -nogpulib the compilation potentially pulls in many libraries from around the system that could vary and cause problems. E.g. a libomptarget.devicertl.a from the system LLVM install or the ROCm device libs from somewhere else. We wanted to have a sanitary environment to run tests in so we elected to disable these libraries manually and then pass them in.

Another thing I've considered is instead of doing -lomptarget.devicertl and -lomptarget we just fish those out of the compiler's resource directory and pass them by name instead. This will pretty much force it to link against only the libraries built with the same LLVM installation. Unfortunately we can't make this fully automatic because we can't use -rpath.

@ergawy
Copy link
Member Author

ergawy commented Jan 2, 2024

What failures is this causing?

These are the failing tests:

Failed Tests (11):
  libomptarget :: amdgcn-amd-amdhsa :: offloading/fortran/basic-target-region-1D-array-section.f90
  libomptarget :: amdgcn-amd-amdhsa :: offloading/fortran/basic-target-region-3D-array-section.f90
  libomptarget :: amdgcn-amd-amdhsa :: offloading/fortran/basic-target-region-3D-array.f90
  libomptarget :: amdgcn-amd-amdhsa :: offloading/fortran/basic_target_region.f90
  libomptarget :: amdgcn-amd-amdhsa :: offloading/fortran/constant-arr-index.f90
  libomptarget :: amdgcn-amd-amdhsa :: offloading/fortran/declare-target-array-in-target-region.f90
  libomptarget :: amdgcn-amd-amdhsa :: offloading/fortran/double-target-call-with-declare-target.f90
  libomptarget :: amdgcn-amd-amdhsa :: offloading/fortran/target-region-implicit-array.f90
  libomptarget :: amdgcn-amd-amdhsa :: offloading/fortran/target_map_common_block.f90
  libomptarget :: amdgcn-amd-amdhsa :: offloading/fortran/target_map_common_block2.f90
  libomptarget :: amdgcn-amd-amdhsa :: offloading/fortran/target_update.f90

And they fail with the following error:

# .---command stderr------------
# | flang-new: error: unknown argument '-nogpulib'; did you mean '-Xclang -nogpulib'?
# `-----------------------------

This was not an oversight, ...

Apologies for assuming that without much background. From the name of the flag, I (possibly incorrectly) assumed that if we pass it to flang, then the device RT lib(s) won't be linked into the final binary which, in turn, will prevent offloading from taking place. That's why, I thought that excluding this flag for the amdgcn-amd-amdhsa target makes sense.

The flag is not supported by flang (as evident from the error message), and there is a PR (#71045) to add it. However, I think adding it might not be the solution for these particular offloading tests.

Please let me know if I missed something.

@jhuber6
Copy link
Contributor

jhuber6 commented Jan 2, 2024

Adding that flag makes sense in Fortran since I'm assuming there will be similar cases where users may wish to disallow the implicit libraries and include paths.

However, if you want a quick fix I'd just suggest making some flag in the lit.cfg that detects if it's Fortran.

@ergawy
Copy link
Member Author

ergawy commented Jan 3, 2024

I will abandon this since @DominikAdamski has a proper solution for the problem in #71045 & #76796.

@ergawy ergawy closed this Jan 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants