[offload] Unit tests failing over "Parameterized test suite ... is defined via TEST_P, but never instantiated"

[mgorny]

With #157253 I was able to use a runtimes build for offload, but then I'm seeing a fair bunch of unittests fail with errors such as:

FAIL: Offload-Unit :: OffloadAPI/./device.unittests/2/4 (4 of 44)
******************** TEST 'Offload-Unit :: OffloadAPI/./device.unittests/2/4' FAILED ********************
Script(shard):
--
GTEST_OUTPUT=json:/var/tmp/portage/llvm-runtimes/offload-22.0.0.9999/work/runtimes_build/offload/unittests/OffloadAPI/./device.unittests-Offload-Unit-209-2-4.json GTEST_SHUFFLE=0 GTEST_TOTAL_SHARDS=4 GTEST_SHARD_INDEX=2 /var/tmp/portage/llvm-runtimes/offload-22.0.0.9999/work/runtimes_build/offload/unittests/OffloadAPI/./device.unittests
--

Script:
--
/var/tmp/portage/llvm-runtimes/offload-22.0.0.9999/work/runtimes_build/offload/unittests/OffloadAPI/./device.unittests --gtest_filter=GoogleTestVerification.UninstantiatedParameterizedTestSuite<olGetDeviceInfoTest>
--
/var/tmp/portage/llvm-runtimes/offload-22.0.0.9999/work/offload/unittests/OffloadAPI/device/olGetDeviceInfo.cpp:14: Failure
Parameterized test suite olGetDeviceInfoTest is defined via TEST_P, but never instantiated. None of the test cases will run. Either no INSTANTIATE_TEST_SUITE_P is provided or the only ones provided expand to nothing.

Ideally, TEST_P definitions should only ever be included as part of binaries that intend to use them. (As opposed to, for example, being placed in a library that may be linked in to get other utilities.)

To suppress this error for this test suite, insert the following line (in a non-header) in the namespace it is defined in:

GTEST_ALLOW_UNINSTANTIATED_PARAMETERIZED_TEST(olGetDeviceInfoTest);


/var/tmp/portage/llvm-runtimes/offload-22.0.0.9999/work/offload/unittests/OffloadAPI/device/olGetDeviceInfo.cpp:14
Parameterized test suite olGetDeviceInfoTest is defined via TEST_P, but never instantiated. None of the test cases will run. Either no INSTANTIATE_TEST_SUITE_P is provided or the only ones provided expand to nothing.

Ideally, TEST_P definitions should only ever be included as part of binaries that intend to use them. (As opposed to, for example, being placed in a library that may be linked in to get other utilities.)

To suppress this error for this test suite, insert the following line (in a non-header) in the namespace it is defined in:

GTEST_ALLOW_UNINSTANTIATED_PARAMETERIZED_TEST(olGetDeviceInfoTest);



********************

and so on. The complete list is:

Failed Tests (39):
  Offload-Unit :: OffloadAPI/./device.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olGetDeviceInfoSizeTest>
  Offload-Unit :: OffloadAPI/./device.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olGetDeviceInfoTest>
  Offload-Unit :: OffloadAPI/./event.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olCreateEventTest>
  Offload-Unit :: OffloadAPI/./event.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olDestroyEventTest>
  Offload-Unit :: OffloadAPI/./event.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olGetEventInfoSizeTest>
  Offload-Unit :: OffloadAPI/./event.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olGetEventInfoTest>
  Offload-Unit :: OffloadAPI/./event.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olSyncEventTest>
  Offload-Unit :: OffloadAPI/./kernel.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olCalculateOptimalOccupancy
Test>
  Offload-Unit :: OffloadAPI/./kernel.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olLaunchKernelFooTest>
  Offload-Unit :: OffloadAPI/./kernel.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olLaunchKernelGlobalCtorTest>
  Offload-Unit :: OffloadAPI/./kernel.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olLaunchKernelGlobalDtorTest>
  Offload-Unit :: OffloadAPI/./kernel.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olLaunchKernelGlobalTest>
  Offload-Unit :: OffloadAPI/./kernel.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olLaunchKernelLocalMemReductionTest>
  Offload-Unit :: OffloadAPI/./kernel.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olLaunchKernelLocalMemStaticTest>
  Offload-Unit :: OffloadAPI/./kernel.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olLaunchKernelLocalMemTest>
  Offload-Unit :: OffloadAPI/./kernel.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olLaunchKernelNoArgsTest>
  Offload-Unit :: OffloadAPI/./memory.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olMemAllocTest>
  Offload-Unit :: OffloadAPI/./memory.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olMemFillTest>
  Offload-Unit :: OffloadAPI/./memory.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olMemFreeTest>
  Offload-Unit :: OffloadAPI/./memory.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olMemcpyGlobalTest>
  Offload-Unit :: OffloadAPI/./memory.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olMemcpyTest>
  Offload-Unit :: OffloadAPI/./platform.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olGetPlatformInfoSizeTest>
  Offload-Unit :: OffloadAPI/./platform.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olGetPlatformInfoTest>
  Offload-Unit :: OffloadAPI/./program.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olCreateProgramTest>
  Offload-Unit :: OffloadAPI/./program.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olDestroyProgramTest>
  Offload-Unit :: OffloadAPI/./queue.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olCreateQueueTest>
  Offload-Unit :: OffloadAPI/./queue.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olDestroyQueueTest>
  Offload-Unit :: OffloadAPI/./queue.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olGetQueueInfoSizeTest>
  Offload-Unit :: OffloadAPI/./queue.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olGetQueueInfoTest>
  Offload-Unit :: OffloadAPI/./queue.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olLaunchHostFunctionKernelTest>
  Offload-Unit :: OffloadAPI/./queue.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olLaunchHostFunctionTest>
  Offload-Unit :: OffloadAPI/./queue.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olSyncQueueTest>
  Offload-Unit :: OffloadAPI/./queue.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olWaitEventsTest>
  Offload-Unit :: OffloadAPI/./symbol.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olGetSymbolGlobalTest>
  Offload-Unit :: OffloadAPI/./symbol.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olGetSymbolInfoGlobalTest>
  Offload-Unit :: OffloadAPI/./symbol.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olGetSymbolInfoKernelTest>
  Offload-Unit :: OffloadAPI/./symbol.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olGetSymbolInfoSizeGlobalTest>
  Offload-Unit :: OffloadAPI/./symbol.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olGetSymbolInfoSizeKernelTest>
  Offload-Unit :: OffloadAPI/./symbol.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olGetSymbolKernelTest>

…leaving 5 tests actually passing.

Originally posted by @mgorny in #156675 (comment)

[mgorny]

Other details from the linked discussion:

CMake invocation:

cmake -C /var/tmp/portage/llvm-runtimes/offload-22.0.0.9999/work/runtimes_build/gentoo_common_config.cmake \
-G Ninja \
-DCMAKE_INSTALL_PREFIX=/usr \
-DLLVM_ROOT=/usr/lib/llvm/22 \
-DLLVM_ENABLE_RUNTIMES=offload \
-DOPENMP_STANDALONE_BUILD=ON \
-OFFLOAD_LIBDIR_SUFFIX=64 \
-DOFFLOAD_INCLUDE_TESTS=yes \
-DLIBOMPTARGET_PLUGINS_TO_BUILD=host;cuda \
-DLIBOMPTARGET_OMPT_SUPPORT=no \
-DLIBOMPTARGET_BUILD_DEVICERTL_BCLIB=TRUE \
-DBUILD_SHARED_LIBS=OFF \
-DFFI_INCLUDE_DIR=/usr/lib64/libffi/include \
-DFFI_LIBRARY_DIR= \
-DFFI_STATIC_LIBRARIES=NO \
-DLIBOMPTARGET_AMDGPU_ARCH=LIBOMPTARGET_AMDGPU_ARCH-NOTFOUND \
-DLIBOMPTARGET_NVPTX_ARCH=LIBOMPTARGET_NVPTX_ARCH-NOTFOUND \
-DOPENMP_LLVM_LIT_EXECUTABLE=/usr/bin/lit \
-DOPENMP_LIT_ARGS=-vv;-j;96 \
-DOPENMP_TEST_C_COMPILER=/usr/lib/ccache/bin/aarch64-unknown-linux-gnu-clang \
-DOPENMP_TEST_CXX_COMPILER=/usr/lib/ccache/bin/aarch64-unknown-linux-gnu-clang++ \
-DOPENMP_TEST_Fortran_COMPILER= \
-DCMAKE_BUILD_TYPE=RelWithDebInfo \
-DCMAKE_TOOLCHAIN_FILE=/var/tmp/portage/llvm-runtimes/offload-22.0.0.9999/work/runtimes_build/gentoo_toolchain.cmake \
/var/tmp/portage/llvm-runtimes/offload-22.0.0.9999/work/runtimes

@jhuber6:

I'm going to assume that this causes some ifdef to be false which never registers the test or something.

-DLIBOMPTARGET_PLUGINS_TO_BUILD=host;cuda

[llvmbot]

@llvm/issue-subscribers-offload

Author: Michał Górny (mgorny)

With #157253 I was able to use a `runtimes` build for offload, but then I'm seeing a fair bunch of unittests fail with errors such as:

FAIL: Offload-Unit :: OffloadAPI/./device.unittests/2/4 (4 of 44)
******************** TEST 'Offload-Unit :: OffloadAPI/./device.unittests/2/4' FAILED ********************
Script(shard):
--
GTEST_OUTPUT=json:/var/tmp/portage/llvm-runtimes/offload-22.0.0.9999/work/runtimes_build/offload/unittests/OffloadAPI/./device.unittests-Offload-Unit-209-2-4.json GTEST_SHUFFLE=0 GTEST_TOTAL_SHARDS=4 GTEST_SHARD_INDEX=2 /var/tmp/portage/llvm-runtimes/offload-22.0.0.9999/work/runtimes_build/offload/unittests/OffloadAPI/./device.unittests
--

Script:
--
/var/tmp/portage/llvm-runtimes/offload-22.0.0.9999/work/runtimes_build/offload/unittests/OffloadAPI/./device.unittests --gtest_filter=GoogleTestVerification.UninstantiatedParameterizedTestSuite<olGetDeviceInfoTest>
--
/var/tmp/portage/llvm-runtimes/offload-22.0.0.9999/work/offload/unittests/OffloadAPI/device/olGetDeviceInfo.cpp:14: Failure
Parameterized test suite olGetDeviceInfoTest is defined via TEST_P, but never instantiated. None of the test cases will run. Either no INSTANTIATE_TEST_SUITE_P is provided or the only ones provided expand to nothing.

Ideally, TEST_P definitions should only ever be included as part of binaries that intend to use them. (As opposed to, for example, being placed in a library that may be linked in to get other utilities.)

To suppress this error for this test suite, insert the following line (in a non-header) in the namespace it is defined in:

GTEST_ALLOW_UNINSTANTIATED_PARAMETERIZED_TEST(olGetDeviceInfoTest);


/var/tmp/portage/llvm-runtimes/offload-22.0.0.9999/work/offload/unittests/OffloadAPI/device/olGetDeviceInfo.cpp:14
Parameterized test suite olGetDeviceInfoTest is defined via TEST_P, but never instantiated. None of the test cases will run. Either no INSTANTIATE_TEST_SUITE_P is provided or the only ones provided expand to nothing.

Ideally, TEST_P definitions should only ever be included as part of binaries that intend to use them. (As opposed to, for example, being placed in a library that may be linked in to get other utilities.)

To suppress this error for this test suite, insert the following line (in a non-header) in the namespace it is defined in:

GTEST_ALLOW_UNINSTANTIATED_PARAMETERIZED_TEST(olGetDeviceInfoTest);



********************

and so on. The complete list is:

Failed Tests (39):
  Offload-Unit :: OffloadAPI/./device.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olGetDeviceInfoSizeTest>
  Offload-Unit :: OffloadAPI/./device.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olGetDeviceInfoTest>
  Offload-Unit :: OffloadAPI/./event.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olCreateEventTest>
  Offload-Unit :: OffloadAPI/./event.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olDestroyEventTest>
  Offload-Unit :: OffloadAPI/./event.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olGetEventInfoSizeTest>
  Offload-Unit :: OffloadAPI/./event.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olGetEventInfoTest>
  Offload-Unit :: OffloadAPI/./event.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olSyncEventTest>
  Offload-Unit :: OffloadAPI/./kernel.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olCalculateOptimalOccupancy
Test>
  Offload-Unit :: OffloadAPI/./kernel.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olLaunchKernelFooTest>
  Offload-Unit :: OffloadAPI/./kernel.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olLaunchKernelGlobalCtorTest>
  Offload-Unit :: OffloadAPI/./kernel.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olLaunchKernelGlobalDtorTest>
  Offload-Unit :: OffloadAPI/./kernel.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olLaunchKernelGlobalTest>
  Offload-Unit :: OffloadAPI/./kernel.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olLaunchKernelLocalMemReductionTest>
  Offload-Unit :: OffloadAPI/./kernel.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olLaunchKernelLocalMemStaticTest>
  Offload-Unit :: OffloadAPI/./kernel.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olLaunchKernelLocalMemTest>
  Offload-Unit :: OffloadAPI/./kernel.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olLaunchKernelNoArgsTest>
  Offload-Unit :: OffloadAPI/./memory.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olMemAllocTest>
  Offload-Unit :: OffloadAPI/./memory.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olMemFillTest>
  Offload-Unit :: OffloadAPI/./memory.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olMemFreeTest>
  Offload-Unit :: OffloadAPI/./memory.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olMemcpyGlobalTest>
  Offload-Unit :: OffloadAPI/./memory.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olMemcpyTest>
  Offload-Unit :: OffloadAPI/./platform.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olGetPlatformInfoSizeTest>
  Offload-Unit :: OffloadAPI/./platform.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olGetPlatformInfoTest>
  Offload-Unit :: OffloadAPI/./program.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olCreateProgramTest>
  Offload-Unit :: OffloadAPI/./program.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olDestroyProgramTest>
  Offload-Unit :: OffloadAPI/./queue.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olCreateQueueTest>
  Offload-Unit :: OffloadAPI/./queue.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olDestroyQueueTest>
  Offload-Unit :: OffloadAPI/./queue.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olGetQueueInfoSizeTest>
  Offload-Unit :: OffloadAPI/./queue.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olGetQueueInfoTest>
  Offload-Unit :: OffloadAPI/./queue.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olLaunchHostFunctionKernelTest>
  Offload-Unit :: OffloadAPI/./queue.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olLaunchHostFunctionTest>
  Offload-Unit :: OffloadAPI/./queue.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olSyncQueueTest>
  Offload-Unit :: OffloadAPI/./queue.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olWaitEventsTest>
  Offload-Unit :: OffloadAPI/./symbol.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olGetSymbolGlobalTest>
  Offload-Unit :: OffloadAPI/./symbol.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olGetSymbolInfoGlobalTest>
  Offload-Unit :: OffloadAPI/./symbol.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olGetSymbolInfoKernelTest>
  Offload-Unit :: OffloadAPI/./symbol.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olGetSymbolInfoSizeGlobalTest>
  Offload-Unit :: OffloadAPI/./symbol.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olGetSymbolInfoSizeKernelTest>
  Offload-Unit :: OffloadAPI/./symbol.unittests/GoogleTestVerification/UninstantiatedParameterizedTestSuite<olGetSymbolKernelTest>

…leaving 5 tests actually passing.

Originally posted by @mgorny in #156675 (comment)

[mgorny]

Oh, for the record, it's quite likely that I've tested that on AArch64. Now I've tried via a regular monorepo build with:

cmake ../llvm -G Ninja -DLLVM_ENABLE_PROJECTS='clang' -DCMAKE_BUILD_TYPE=MinSizeRel \
  -DLLVM_CCACHE_BUILD=ON -DLLVM_BUILD_LLVM_DYLIB=ON -DLLVM_LINK_LLVM_DYLIB=ON \
  -DCLANG_LINK_CLANG_DYLIB=ON -DLLVM_TARGETS_TO_BUILD=AArch64 \
  -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_ENABLE_RUNTIMES='openmp;offload'
ninja
cd runtimes/runtimes-bins
ninja check-offload-unit

and it's failing the same way, even though:

LIBOMPTARGET_DLOPEN_PLUGINS:STRING=amdgpu;cuda;host
LIBOMPTARGET_PLUGINS_TO_BUILD:STRING=all

[RossBrunton]

Were these tests ran on a device where there was no valid device (i.e. no Nvidia or AMD devices, or the drivers were not installed or whatever)? This error seems to happen if there are no devices found, so it can't generate the parametrised tests.

[mgorny]

Yes, indeed this is in a container with no GPU.

[RossBrunton]

#157417 Might fix it.

[mgorny]

Fixed by #157417.