[flang][openmp] undefined symbols with device runtime on amdgpu

[VeeEM]

I'm getting some undefined symbols with the flang offload runtime if I try to assign to an array in a target region. Array assignments in target regions have worked previously.

Here's a reproducer

program assign
implicit none

integer, dimension(10) :: xs

!$omp target map(xs)
xs = 1
!$omp end target

print *, xs

end program

Trying to build it with
$ flang -fopenmp --offload-arch=gfx1100 assign.f90 -L$RUNTIME_DIR
outputs

flang-21: warning: OpenMP support in flang is still experimental [-Wexperimental-option]
flang-21: warning: OpenMP support in flang is still experimental [-Wexperimental-option]
flang-21: warning: OpenMP support in flang is still experimental [-Wexperimental-option]
ld.lld: error: undefined symbol: std::in_place_index<0ul>
>>> referenced by /tmp/a.out.amdgcn.gfx1100-542f42.img.lto.o:(void std::_Construct<std::__detail::__variant::_Uninitialized<Fortran::runtime::AssignTicket, true>, std::in_place_index_t<0ul> const&, Fortran::runtime::Descriptor&, Fortran::runtime::Descriptor const&, int&, void* (*&)(void*, void const*, unsigned long), Fortran::runtime::typeInfo::DerivedType const*&>(std::__detail::__variant::_Uninitialized<Fortran::runtime::AssignTicket, true>*, std::in_place_index_t<0ul> const&, Fortran::runtime::Descriptor&, Fortran::runtime::Descriptor const&, int&, void* (*&)(void*, void const*, unsigned long), Fortran::runtime::typeInfo::DerivedType const*&))
>>> referenced by /tmp/a.out.amdgcn.gfx1100-542f42.img.lto.o:(void std::_Construct<std::__detail::__variant::_Uninitialized<Fortran::runtime::AssignTicket, true>, std::in_place_index_t<0ul> const&, Fortran::runtime::Descriptor&, Fortran::runtime::Descriptor const&, int&, void* (*&)(void*, void const*, unsigned long), Fortran::runtime::typeInfo::DerivedType const*&>(std::__detail::__variant::_Uninitialized<Fortran::runtime::AssignTicket, true>*, std::in_place_index_t<0ul> const&, Fortran::runtime::Descriptor&, Fortran::runtime::Descriptor const&, int&, void* (*&)(void*, void const*, unsigned long), Fortran::runtime::typeInfo::DerivedType const*&))
>>> referenced by /tmp/a.out.amdgcn.gfx1100-542f42.img.lto.o:(void std::_Construct<std::__detail::__variant::_Uninitialized<Fortran::runtime::FinalizeTicket, true>, std::in_place_index_t<0ul> const&, Fortran::runtime::Descriptor const&, Fortran::runtime::typeInfo::DerivedType const&>(std::__detail::__variant::_Uninitialized<Fortran::runtime::FinalizeTicket, true>*, std::in_place_index_t<0ul> const&, Fortran::runtime::Descriptor const&, Fortran::runtime::typeInfo::DerivedType const&))
>>> referenced 11 more times

ld.lld: error: undefined symbol: Fortran::runtime::DerivedAssignTicket<true>::Begin(Fortran::runtime::WorkQueue&)
>>> referenced by /tmp/a.out.amdgcn.gfx1100-542f42.img.lto.o:(_ZZN7Fortran7runtime6Ticket8ContinueERNS0_9WorkQueueEENK3$_0clINS0_19DerivedAssignTicketILb1EEEEEDaRT_)
>>> referenced by /tmp/a.out.amdgcn.gfx1100-542f42.img.lto.o:(_ZZN7Fortran7runtime6Ticket8ContinueERNS0_9WorkQueueEENK3$_0clINS0_19DerivedAssignTicketILb1EEEEEDaRT_)

ld.lld: error: undefined symbol: Fortran::runtime::io::descr::DescriptorIoTicket<(Fortran::runtime::io::Direction)0>::Begin(Fortran::runtime::WorkQueue&)
>>> referenced by /tmp/a.out.amdgcn.gfx1100-542f42.img.lto.o:(_ZZN7Fortran7runtime6Ticket8ContinueERNS0_9WorkQueueEENK3$_0clINS0_2io5descr18DescriptorIoTicketILNS6_9DirectionE0EEEEEDaRT_)
>>> referenced by /tmp/a.out.amdgcn.gfx1100-542f42.img.lto.o:(_ZZN7Fortran7runtime6Ticket8ContinueERNS0_9WorkQueueEENK3$_0clINS0_2io5descr18DescriptorIoTicketILNS6_9DirectionE0EEEEEDaRT_)

ld.lld: error: undefined symbol: Fortran::runtime::io::descr::DescriptorIoTicket<(Fortran::runtime::io::Direction)1>::Begin(Fortran::runtime::WorkQueue&)
>>> referenced by /tmp/a.out.amdgcn.gfx1100-542f42.img.lto.o:(_ZZN7Fortran7runtime6Ticket8ContinueERNS0_9WorkQueueEENK3$_0clINS0_2io5descr18DescriptorIoTicketILNS6_9DirectionE1EEEEEDaRT_)
>>> referenced by /tmp/a.out.amdgcn.gfx1100-542f42.img.lto.o:(_ZZN7Fortran7runtime6Ticket8ContinueERNS0_9WorkQueueEENK3$_0clINS0_2io5descr18DescriptorIoTicketILNS6_9DirectionE1EEEEEDaRT_)

ld.lld: error: undefined symbol: Fortran::runtime::DerivedAssignTicket<false>::Begin(Fortran::runtime::WorkQueue&)
>>> referenced by /tmp/a.out.amdgcn.gfx1100-542f42.img.lto.o:(_ZZN7Fortran7runtime6Ticket8ContinueERNS0_9WorkQueueEENK3$_0clINS0_19DerivedAssignTicketILb0EEEEEDaRT_)
>>> referenced by /tmp/a.out.amdgcn.gfx1100-542f42.img.lto.o:(_ZZN7Fortran7runtime6Ticket8ContinueERNS0_9WorkQueueEENK3$_0clINS0_19DerivedAssignTicketILb0EEEEEDaRT_)

ld.lld: error: undefined symbol: Fortran::runtime::DerivedAssignTicket<true>::Continue(Fortran::runtime::WorkQueue&)
>>> referenced by /tmp/a.out.amdgcn.gfx1100-542f42.img.lto.o:(_ZZN7Fortran7runtime6Ticket8ContinueERNS0_9WorkQueueEENK3$_1clINS0_19DerivedAssignTicketILb1EEEEEDaRT_)
>>> referenced by /tmp/a.out.amdgcn.gfx1100-542f42.img.lto.o:(_ZZN7Fortran7runtime6Ticket8ContinueERNS0_9WorkQueueEENK3$_1clINS0_19DerivedAssignTicketILb1EEEEEDaRT_)

ld.lld: error: undefined symbol: Fortran::runtime::io::descr::DescriptorIoTicket<(Fortran::runtime::io::Direction)0>::Continue(Fortran::runtime::WorkQueue&)
>>> referenced by /tmp/a.out.amdgcn.gfx1100-542f42.img.lto.o:(_ZZN7Fortran7runtime6Ticket8ContinueERNS0_9WorkQueueEENK3$_1clINS0_2io5descr18DescriptorIoTicketILNS6_9DirectionE0EEEEEDaRT_)
>>> referenced by /tmp/a.out.amdgcn.gfx1100-542f42.img.lto.o:(_ZZN7Fortran7runtime6Ticket8ContinueERNS0_9WorkQueueEENK3$_1clINS0_2io5descr18DescriptorIoTicketILNS6_9DirectionE0EEEEEDaRT_)

ld.lld: error: undefined symbol: Fortran::runtime::io::descr::DescriptorIoTicket<(Fortran::runtime::io::Direction)1>::Continue(Fortran::runtime::WorkQueue&)
>>> referenced by /tmp/a.out.amdgcn.gfx1100-542f42.img.lto.o:(_ZZN7Fortran7runtime6Ticket8ContinueERNS0_9WorkQueueEENK3$_1clINS0_2io5descr18DescriptorIoTicketILNS6_9DirectionE1EEEEEDaRT_)
>>> referenced by /tmp/a.out.amdgcn.gfx1100-542f42.img.lto.o:(_ZZN7Fortran7runtime6Ticket8ContinueERNS0_9WorkQueueEENK3$_1clINS0_2io5descr18DescriptorIoTicketILNS6_9DirectionE1EEEEEDaRT_)

ld.lld: error: undefined symbol: Fortran::runtime::io::descr::DerivedIoTicket<(Fortran::runtime::io::Direction)0>::Continue(Fortran::runtime::WorkQueue&)
>>> referenced by /tmp/a.out.amdgcn.gfx1100-542f42.img.lto.o:(_ZZN7Fortran7runtime6Ticket8ContinueERNS0_9WorkQueueEENK3$_1clINS0_2io5descr15DerivedIoTicketILNS6_9DirectionE0EEEEEDaRT_)
>>> referenced by /tmp/a.out.amdgcn.gfx1100-542f42.img.lto.o:(_ZZN7Fortran7runtime6Ticket8ContinueERNS0_9WorkQueueEENK3$_1clINS0_2io5descr15DerivedIoTicketILNS6_9DirectionE0EEEEEDaRT_)

ld.lld: error: undefined symbol: Fortran::runtime::io::descr::DerivedIoTicket<(Fortran::runtime::io::Direction)1>::Continue(Fortran::runtime::WorkQueue&)
>>> referenced by /tmp/a.out.amdgcn.gfx1100-542f42.img.lto.o:(_ZZN7Fortran7runtime6Ticket8ContinueERNS0_9WorkQueueEENK3$_1clINS0_2io5descr15DerivedIoTicketILNS6_9DirectionE1EEEEEDaRT_)
>>> referenced by /tmp/a.out.amdgcn.gfx1100-542f42.img.lto.o:(_ZZN7Fortran7runtime6Ticket8ContinueERNS0_9WorkQueueEENK3$_1clINS0_2io5descr15DerivedIoTicketILNS6_9DirectionE1EEEEEDaRT_)

ld.lld: error: undefined symbol: Fortran::runtime::DerivedAssignTicket<false>::Continue(Fortran::runtime::WorkQueue&)
>>> referenced by /tmp/a.out.amdgcn.gfx1100-542f42.img.lto.o:(_ZZN7Fortran7runtime6Ticket8ContinueERNS0_9WorkQueueEENK3$_1clINS0_19DerivedAssignTicketILb0EEEEEDaRT_)
>>> referenced by /tmp/a.out.amdgcn.gfx1100-542f42.img.lto.o:(_ZZN7Fortran7runtime6Ticket8ContinueERNS0_9WorkQueueEENK3$_1clINS0_19DerivedAssignTicketILb0EEEEEDaRT_)
clang: error: ld.lld command failed with exit code 1 (use -v to see invocation)
/home/acc/llvm-stuff/llvm_upstream/install/bin/clang-linker-wrapper: error: 'clang' failed
flang-21: error: linker command failed with exit code 1 (use -v to see invocation)

I'm on commit 163871c and have built clang and flang with these configurations:

cmake \
  -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_INSTALL_PREFIX=$INSTALLDIR \
  -DCMAKE_CXX_STANDARD=17 \
  -DCMAKE_EXPORT_COMPILE_COMMANDS=ON \
  -DCMAKE_CXX_LINK_FLAGS="-Wl,-rpath,$LD_LIBRARY_PATH" \
  -DLLVM_ENABLE_ASSERTIONS=ON \
  -DLLVM_TARGETS_TO_BUILD='X86;AMDGPU' \
  -DLLVM_LIT_ARGS=-v \
  -DLLVM_ENABLE_PROJECTS='clang;mlir;flang;lld' \
  -DLLVM_ENABLE_RUNTIMES='compiler-rt;openmp;offload;flang-rt' \
  -DRUNTIMES_amdgcn-amd-amdhsa_LLVM_ENABLE_RUNTIMES=libc \
  -DLLVM_RUNTIME_TARGETS='default;amdgcn-amd-amdhsa' \
  -DLLVM_CCACHE_BUILD=ON \
  ../llvm-project/llvm

And the offload runtime with these:

cmake  \
  -DLLVM_ENABLE_RUNTIMES=flang-rt \
  -DFLANG_RT_EXPERIMENTAL_OFFLOAD_SUPPORT="OpenMP" \
  -DCMAKE_C_COMPILER=clang \
  -DCMAKE_CXX_COMPILER=clang++ \
  -DFLANG_RT_DEVICE_ARCHITECTURES='gfx1100' \
  -DFLANG_RT_INCLUDE_TESTS=OFF \
  ../llvm-project/runtimes/

[VeeEM]

Seems to work if I revert 2bf3cca.

[vzakhari]

@VeeEM do you know what these callers are?

>>> referenced by /tmp/a.out.amdgcn.gfx1100-542f42.img.lto.o:(_ZZN7Fortran7runtime6Ticket8ContinueERNS0_9WorkQueueEENK3$_0clINS0_19DerivedAssignTicketILb1EEEEEDaRT_)
>>> referenced by /tmp/a.out.amdgcn.gfx1100-542f42.img.lto.o:(_ZZN7Fortran7runtime6Ticket8ContinueERNS0_9WorkQueueEENK3$_0clINS0_19DerivedAssignTicketILb1EEEEEDaRT_)

I am having hard time demangling these names.

[VeeEM]

@vzakhari I think those are the lambdas in Ticket::Continue

llvm-project/flang-rt/lib/runtime/work-queue.cpp

Line 39 in e4d8e06

RT_API_ATTRS int Ticket::Continue(WorkQueue &workQueue) {

This is what boost demangle gives me for that symbol

auto Fortran::runtime::Ticket::Continue(Fortran::runtime::WorkQueue&)::$_0::operator()<Fortran::runtime::DerivedAssignTicket<true> >(Fortran::runtime::DerivedAssignTicket<true>&) const

[vzakhari]

Thank you!

Okay, so the lambdas are calling the methods that are defined in another module assign.cpp, but I do see that those methods are properly marked for OpenMP offload in assign.cpp:

RT_OFFLOAD_API_GROUP_BEGIN
...
template <bool IS_COMPONENTWISE>
RT_API_ATTRS int DerivedAssignTicket<IS_COMPONENTWISE>::Begin(
    WorkQueue &workQueue) {
...
}
template RT_API_ATTRS int DerivedAssignTicket<false>::Begin(WorkQueue &);
template RT_API_ATTRS int DerivedAssignTicket<true>::Begin(WorkQueue &);
...
RT_OFFLOAD_API_GROUP_END

Those RT_OFFLOAD macros expand into _Pragma("omp declare target") and _Pragma("omp end declare target"), so they should be available in the device code for the external linkage.

I am not sure if it will be easy to reproduce this on my side (I suppose, I will need some AMD tools for that). Could you please verify that the missing methods are indeed undefined in the assign.cpp.o object file?

[VeeEM]

The symbols seem to be defined in assign.cpp.o

$ llvm-nm --demangle assign.cpp.o | grep 'Fortran::runtime::DerivedAssignTicket<true>::Begin(Fortran::runtime::WorkQueue&)'
0000000000000000 W Fortran::runtime::DerivedAssignTicket<true>::Begin(Fortran::runtime::WorkQueue&)
$ llvm-nm --demangle work-queue.cpp.o | grep 'Fortran::runtime::DerivedAssignTicket<true>::Begin(Fortran::runtime::WorkQueue&)'
                 U Fortran::runtime::DerivedAssignTicket<true>::Begin(Fortran::runtime::WorkQueue&)

but missing from the device image

$ llvm-objcopy --dump-section=.llvm.offloading=work-queue-device.cpp.o work-queue.cpp.o
$ dd if=./work-queue-device.cpp.o of=./work-queue-device.cpp.o.elf skip=144 iflag=skip_bytes
$ llvm-objcopy --dump-section=.llvm.offloading=assign-device.cpp.o assign.cpp.o
$ dd if=./assign-device.cpp.o of=./assign-device.cpp.o.elf skip=144 iflag=skip_bytes

$ llvm-nm --demangle work-queue-device.cpp.o.elf | grep 'Fortran::runtime::DerivedAssignTicket<true>::Begin(Fortran::runtime::WorkQueue&)'
         U Fortran::runtime::DerivedAssignTicket<true>::Begin(Fortran::runtime::WorkQueue&)
$ llvm-nm --demangle assign-device.cpp.o.elf | grep 'Fortran::runtime::DerivedAssignTicket<true>::Begin(Fortran::runtime::WorkQueue&)'

Not sure what's going on there, but could it be that std::variant is causing problems? The CUDA build uses libcudacxx while the OpenMP build uses the standard host C++ library. std::in_place_index<0ul> being one of the undefined symbols hints at this too.

[VeeEM]

I haven't been using OpenMP offloading with flang in an Nvidia environment so I'm not sure what the general state of things is, but looks like this same problem occurs there too. I now tested the same reproducer on a machine with an Nvidia GPU, just switching the architecture flags where appropriate, and this is what I get:

$ flang assign.f90 -fopenmp --offload-arch=sm_89 -L$RUNTIME_DIR
nvlink error   : Undefined reference to '_ZN7Fortran7runtime19DerivedAssignTicketILb0EE8ContinueERNS0_9WorkQueueE' in '/tmp/a.out.nvptx64.sm_89-5b1391-0ae376.cubin'
nvlink error   : Undefined reference to '_ZN7Fortran7runtime2io5descr15DerivedIoTicketILNS1_9DirectionE1EE8ContinueERNS0_9WorkQueueE' in '/tmp/a.out.nvptx64.sm_89-5b1391-0ae376.cubin'
nvlink error   : Undefined reference to '_ZN7Fortran7runtime2io5descr15DerivedIoTicketILNS1_9DirectionE0EE8ContinueERNS0_9WorkQueueE' in '/tmp/a.out.nvptx64.sm_89-5b1391-0ae376.cubin'
nvlink error   : Undefined reference to '_ZN7Fortran7runtime2io5descr18DescriptorIoTicketILNS1_9DirectionE1EE8ContinueERNS0_9WorkQueueE' in '/tmp/a.out.nvptx64.sm_89-5b1391-0ae376.cubin'
nvlink error   : Undefined reference to '_ZN7Fortran7runtime2io5descr18DescriptorIoTicketILNS1_9DirectionE0EE8ContinueERNS0_9WorkQueueE' in '/tmp/a.out.nvptx64.sm_89-5b1391-0ae376.cubin'
nvlink error   : Undefined reference to '_ZN7Fortran7runtime19DerivedAssignTicketILb1EE8ContinueERNS0_9WorkQueueE' in '/tmp/a.out.nvptx64.sm_89-5b1391-0ae376.cubin'
nvlink error   : Undefined reference to '_ZN7Fortran7runtime19DerivedAssignTicketILb0EE5BeginERNS0_9WorkQueueE' in '/tmp/a.out.nvptx64.sm_89-5b1391-0ae376.cubin'
nvlink error   : Undefined reference to '_ZN7Fortran7runtime2io5descr18DescriptorIoTicketILNS1_9DirectionE1EE5BeginERNS0_9WorkQueueE' in '/tmp/a.out.nvptx64.sm_89-5b1391-0ae376.cubin'
nvlink error   : Undefined reference to '_ZN7Fortran7runtime2io5descr18DescriptorIoTicketILNS1_9DirectionE0EE5BeginERNS0_9WorkQueueE' in '/tmp/a.out.nvptx64.sm_89-5b1391-0ae376.cubin'
nvlink error   : Undefined reference to '_ZN7Fortran7runtime19DerivedAssignTicketILb1EE5BeginERNS0_9WorkQueueE' in '/tmp/a.out.nvptx64.sm_89-5b1391-0ae376.cubin'
nvlink error   : Undefined reference to '_ZSt14in_place_indexILm0EE' in '/tmp/a.out.nvptx64.sm_89-5b1391-0ae376.cubin'
/home/acc/Documents/flang_root/install/bin/clang-nvlink-wrapper: error: 'nvlink' failed
clang: error: nvlink command failed with exit code 1 (use -v to see invocation)
/home/acc/Documents/flang_root/install/bin/clang-linker-wrapper: error: 'clang' failed
flang-21: error: linker command failed with exit code 1 (use -v to see invocation)

[hakostra]

I have flang built for offload to Nvidia:

flang version 21.0.0git (https://github.com/llvm/llvm-project.git 7354123c34e658e990559a36b1ac7eb0b671e317)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/llvm/llvm-project/install/bin

And the example in the original post compiles and runs just fine.

[VeeEM]

Interesting, how do you configure the runtime? This is what I used:

cmake -DLLVM_ENABLE_RUNTIMES=flang-rt \
-DFLANG_RT_EXPERIMENTAL_OFFLOAD_SUPPORT="OpenMP" \
-DCMAKE_Fortran_COMPILER=flang  \
-DCMAKE_C_COMPILER=clang \
-DCMAKE_CXX_COMPILER=clang++ \
-DFLANG_RT_DEVICE_ARCHITECTURES=sm_89 \
-DFLANG_RT_INCLUDE_TESTS=OFF \
../llvm-project/runtimes/

[hakostra]

I build LLVM with this CMake command:

cmake -GNinja \
    -DCMAKE_BUILD_TYPE="Release" \
    -DCMAKE_INSTALL_PREFIX="/opt/llvm/llvm-project/install" \
    -DCMAKE_CUDA_COMPILER=${CUDACXX} \
    -DCMAKE_CUDA_HOST_COMPILER="clang++-19" \
    -DCMAKE_CUDA_ARCHITECTURES="89;120" \
    -DLLVM_ENABLE_PROJECTS="clang;lld;flang;lldb" \
    -DLLVM_ENABLE_RUNTIMES="compiler-rt;openmp;offload;flang-rt" \
    -DLLVM_TARGETS_TO_BUILD="X86;NVPTX" \
    -DLLVM_ENABLE_RTTI="ON" \
    -DOPENMP_ENABLE_LIBOMPTARGET="ON" \
    -DFLANG_RT_EXPERIMENTAL_OFFLOAD_SUPPORT="OpenMP" \
    -DFLANG_RT_DEVICE_ARCHITECTURES="sm_89;sm_120" \
    -DFLANG_EXPERIMENTAL_OMP_OFFLOAD_BUILD="host_device" \
    ../llvm

My experience is that the OpenMP offload features in flang is getting very, very good now.

[VeeEM]

Well, I'm not having any luck getting assign to work with offloading.

I looked a bit more into what is happening here and it seems that only std::in_place_index being undefined is related to the use of std::variant.

It's a bit wonky how templates work with declare target. Take this example code:

template <bool B>
class Foo {
  public:
  int GetNumber();
};

struct Bar {
  Foo<true> u;
  Foo<false> y;
};

_Pragma("omp declare target")

template <bool B>
int Foo<B>::GetNumber() {
  if (B) return 1;
  else return 2;
}

template int Foo<true>::GetNumber();
template int Foo<false>::GetNumber();

_Pragma("omp end declare target")

Here the use of Foo and Foo in the definition of Bar causes the GetNumber functions to not be generated for the device. If we remove Bar or put declare target pragmas around the declaration of Foo, then the functions are compiled for the device. Also if explicit specializations are used like this:

template<>
int Foo<true>::GetNumber() {
  return 1;
}
template<>
int Foo<false>::GetNumber() {
  return 2;
}

the functions will be generated for the device.

That is with clang++ -fopenmp --offload-arch=sm_89 -O3 -c. Every compiler seems to produce different results for this code. I tested it also with gcc and nvc++. Clang requires the class declaration to be in a declare target redion, gcc always generates the device code, and nvc++ only generates the device functions if the explicit specialization form is used.

It is possible to get those missing flang runtime functions generated on the device by adding more declare target pragmas into header files where needed, but this still leaves std::in_place_index undefined in the device image.