Changes needed for the STDdata for Spack Package to Work

[kaanolgu]

The Spack couldn't link the TBB to the package but with this small fix, it is now building the STDdata with TBB and the performance is comparable to the results from CMake

[tom91136]

Can you check if this conflicts with the develop branch and switch base to that?

[tom91136]

Cool, let's see if CI is green.

[kaanolgu]

Cool, let's see if CI is green.

NVHPC fails in some versions of CMake and it succeeds with some versions ?

[tom91136]

Is this supposed to be inverted? Currently it's only gonna find TBB if USE_TBB is disabled.

[tom91136]

The CI failure is because older CMake versions probably don't have good support for find_package(TBB REQUIRED) but I think that can be worked around by just specifying -ltbb or equivalent.

[kaanolgu]

I discovered that in order to spack installed tbb work, it needs that find_package without it it doesn't work and fails to build
I set USE_TBB=OFF and then provide the TBB_ROOT value to be picked from the spack installed dependency location

[tom91136]

I see. So I'm a bit hesitant to merge this because std-* models needing TBB is an implementation detail for libstdc++. For the newer libc++ or Intel's oneDPL fork, there are options for using OpenMP and other backend, etc.
How about having a -DFIND_TBB instead that enables find_package(TBB REQUIRED) and registers the libraries?
And in Spack, you just do -DFIND_TBB=ON.

[kaanolgu]

Sure, ideally that should work since it will be set, I will check it in the evening and modify the PR

[kaanolgu]

I made the following changes but CMake gives error :

1. Adding the following to the main CMakeLists.txt :

option(FIND_TBB "Enable the oneTBB library for *supported* models. Enabling this on models that
                don't explicitly link against TBB is a no-op, see description of your selected
                model on how this is used." OFF)

2. Adding this option to the src/std-data/model.cmake :

register_flag_optional(FIND_TBB
        "This option makes sure that while installing with Spack, the intel-tbb package is automatically picked up by Spack. Advised to use while using Spack package.
        
        The TBB_ROOT also needs to be set from the Spack package with this option"
        "OFF")
register_flag_optional(SPACK_TBB
        "This option makes sure that while installing with Spack, the intel-tbb package is automatically picked up by Spack. Advised to use while using Spack package.
        
        The TBB_ROOT also needs to be set from the Spack package with this option"
        "")

3. Modify the macro to the following :

macro(setup)
    set(CMAKE_CXX_STANDARD 17)
    if (NVHPC_OFFLOAD)
        set(NVHPC_FLAGS -stdpar -gpu=${NVHPC_OFFLOAD})
        # propagate flags to linker so that it links with the gpu stuff as well
        register_append_cxx_flags(ANY ${NVHPC_FLAGS})
        register_append_link_flags(${NVHPC_FLAGS})
    endif ()
    if (FIND_TBB)
        find_package(TBB REQUIRED)
        set(TBB_ROOT "${SPACK_TBB}")
        register_link_library(TBB::tbb)
    endif ()
    if (USE_TBB)
        register_link_library(TBB::tbb)
    endif ()
    if (USE_ONEDPL)
        register_definitions(USE_ONEDPL)
        register_link_library(oneDPL)
    endif ()
endmacro()

4. Setting the following flags from Spack :

        if "+stddata" in self.spec:  #  spec_string.startswith(tuple(std_list)):
            args.append("-DCMAKE_CXX_COMPILER=" + spack_cxx)
            args.append("-DFIND_TBB=ON")
            args.append( "-DSPACK_TBB=" + self.spec["intel-oneapi-tbb"].prefix + "/tbb/latest/")

Gives me the following error message without any context :

$ spack install babelstream@test_stddata%gcc@13.1.0 +stddata 
[+] /lustre/home/br-kolgu/spack/opt/spack/cray-rhel8-cascadelake/gcc-13.1.0/ncurses-6.4-c5teo32kus5fe2foee7k3tguawovvj4g
[+] /lustre/home/br-kolgu/spack/opt/spack/cray-rhel8-cascadelake/gcc-13.1.0/zlib-1.2.13-rpt3qsbd22sq2qmxtdgilvxvpkugm5jy
[+] /lustre/home/br-kolgu/spack/opt/spack/cray-rhel8-cascadelake/gcc-13.1.0/gmake-4.4.1-x6dbb4hm4lbnsplaaqjiik227tv4st6k
[+] /lustre/home/br-kolgu/spack/opt/spack/cray-rhel8-cascadelake/gcc-13.1.0/intel-oneapi-tbb-2021.9.0-bmhubsi6jb3qxfog4iti2hln7qxugrod
[+] /lustre/home/br-kolgu/spack/opt/spack/cray-rhel8-cascadelake/gcc-13.1.0/openssl-1.1.1t-ukzzukl46wjpt3n7zkmkvnqyx6i3xa6u
[+] /lustre/home/br-kolgu/spack/opt/spack/cray-rhel8-cascadelake/gcc-13.1.0/cmake-3.26.3-tbebl5yaatq5iltza7doftyhl2htvh7s
==> Installing babelstream-test_stddata-htgfjuodtvej245ebqgwkgfynzhesfis
==> No binary for babelstream-test_stddata-htgfjuodtvej245ebqgwkgfynzhesfis found: installing from source
==> No patches needed for babelstream
==> babelstream: Executing phase: 'cmake'
==> Error: ProcessError: Command exited with status -4:
    '/lustre/home/br-kolgu/spack/opt/spack/cray-rhel8-cascadelake/gcc-13.1.0/cmake-3.26.3-tbebl5yaatq5iltza7doftyhl2htvh7s/bin/cmake' '-G' 'Unix Makefiles' '-DCMAKE_INSTALL_PREFIX:STRING=/lustre/home/br-kolgu/spack/opt/spack/cray-rhel8-cascadelake/gcc-13.1.0/babelstream-test_stddata-htgfjuodtvej245ebqgwkgfynzhesfis' '-DCMAKE_BUILD_TYPE:STRING=Release' '-DBUILD_TESTING:BOOL=OFF' '-DCMAKE_INTERPROCEDURAL_OPTIMIZATION:BOOL=OFF' '-DCMAKE_VERBOSE_MAKEFILE:BOOL=ON' '-DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=ON' '-DCMAKE_INSTALL_RPATH:STRING=/lustre/home/br-kolgu/spack/opt/spack/cray-rhel8-cascadelake/gcc-13.1.0/babelstream-test_stddata-htgfjuodtvej245ebqgwkgfynzhesfis/lib;/lustre/home/br-kolgu/spack/opt/spack/cray-rhel8-cascadelake/gcc-13.1.0/babelstream-test_stddata-htgfjuodtvej245ebqgwkgfynzhesfis/lib64;/lustre/home/br-kolgu/spack/opt/spack/cray-rhel8-cascadelake/gcc-13.1.0/intel-oneapi-tbb-2021.9.0-bmhubsi6jb3qxfog4iti2hln7qxugrod/lib;None' '-DCMAKE_PREFIX_PATH:STRING=/lustre/home/br-kolgu/spack/opt/spack/cray-rhel8-cascadelake/gcc-13.1.0/intel-oneapi-tbb-2021.9.0-bmhubsi6jb3qxfog4iti2hln7qxugrod;/lustre/home/br-kolgu/spack/opt/spack/cray-rhel8-cascadelake/gcc-13.1.0/gmake-4.4.1-x6dbb4hm4lbnsplaaqjiik227tv4st6k;/lustre/home/br-kolgu/spack/opt/spack/cray-rhel8-cascadelake/gcc-13.1.0/cmake-3.26.3-tbebl5yaatq5iltza7doftyhl2htvh7s' '-DMODEL=std-data' '-DCMAKE_CXX_COMPILER=/lustre/home/br-kolgu/spack/lib/spack/env/gcc/g++' '-DFIND_TBB=ON' '-DSPACK_TBB=/lustre/home/br-kolgu/spack/opt/spack/cray-rhel8-cascadelake/gcc-13.1.0/intel-oneapi-tbb-2021.9.0-bmhubsi6jb3qxfog4iti2hln7qxugrod/tbb/latest/' '/tmp/br-kolgu/spack-stage/spack-stage-babelstream-test_stddata-htgfjuodtvej245ebqgwkgfynzhesfis/spack-src'

[kaanolgu]

@tom91136 @tomdeakin :
It's been a long time now but I found the solution. It requires the following flag for the Spack package to make sure that CMake succesfully picks TBB and executes the register_link_library(TBB::tbb) :

args.append("-DCMAKE_PREFIX_PATH=" + self.spec["intel-tbb"].prefix + "/tbb/latest/")

For the future: I did test with Isambard-Phase3 with Milan CPU and works as expected with gcc@12.2.0 std-data
I will update the spack/spack#41019 with this change

EDIT: no need for any change in BabelStream package
EDIT 2 : Actually, further tests showed me that the intel-tbb path is automatically included in the CMAKE_PREFIX_PATH when dependency is added the key part is -DUSE_TBB=ON is not working since I think it is TBB:tbb ( I will double check this later) but adding -DCXX_EXTRA_FLAGS=-ltbb solved the issue