[HIPIFY] Changing compiler and cuda environment related section to ROCm?

[klausbu]

I am hipifying RapidCFD, an on GPU version of OpenFOAM.

How to modify, the following, compiler and cuda environment related section in the settings file to use hipcc and the Thrust implementation provided by ROCm:

# Compiler settings
# ~~~~~~~~~~~~~~~~~
unset gcc_version gmp_version mpfr_version mpc_version
unset MPFR_ARCH_PATH GMP_ARCH_PATH

# Location of compiler installation
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
if [ -z "$foamCompiler" ]
then
    foamCompiler=system
    echo "Warning in $WM_PROJECT_DIR/etc/config/settings.sh:" 1>&2
    echo "    foamCompiler not set, using '$foamCompiler'" 1>&2
fi

case "${foamCompiler}" in
OpenFOAM | ThirdParty)
    case "$WM_COMPILER" in
    Gcc | Gcc++0x | Gcc48 | Gcc48++0x)
        gcc_version=gcc-4.8.2
        gmp_version=gmp-5.1.2
        mpfr_version=mpfr-3.1.2
        mpc_version=mpc-1.0.1
        ;;
    Gcc49 | Gcc49++0x)
        gcc_version=gcc-4.9.0
        gmp_version=gmp-5.1.2
        mpfr_version=mpfr-3.1.2
        mpc_version=mpc-1.0.1
        ;;
    Clang)
        # using clang - not gcc
        export WM_CC='clang'
        export WM_CXX='clang++'
        clang_version=llvm-3.4.2
        ;;
    *)
        echo 1>&2
        echo "Warning in $WM_PROJECT_DIR/etc/config/settings.sh:" 1>&2
        echo "    Unknown OpenFOAM compiler type '$WM_COMPILER'" 1>&2
        echo "    Please check your settings" 1>&2
        echo 1>&2
        ;;
    esac

    # optional configuration tweaks:
    _foamSource `$WM_PROJECT_DIR/bin/foamEtcFile config/compiler.sh`

    if [ -n "$gcc_version" ]
    then
        gccDir=$WM_THIRD_PARTY_DIR/platforms/$WM_ARCH$WM_COMPILER_ARCH/$gcc_version
        gmpDir=$WM_THIRD_PARTY_DIR/platforms/$WM_ARCH$WM_COMPILER_ARCH/$gmp_version
        mpfrDir=$WM_THIRD_PARTY_DIR/platforms/$WM_ARCH$WM_COMPILER_ARCH/$mpfr_version
        mpcDir=$WM_THIRD_PARTY_DIR/platforms/$WM_ARCH$WM_COMPILER_ARCH/$mpc_version

        # Check that the compiler directory can be found
        [ -d "$gccDir" ] || {
            echo 1>&2
            echo "Warning in $WM_PROJECT_DIR/etc/config/settings.sh:" 1>&2
            echo "    Cannot find $gccDir installation." 1>&2
            echo "    Please install this compiler version or if you wish to use the system compiler," 1>&2
            echo "    change the 'foamCompiler' setting to 'system'" 1>&2
            echo
        }

        _foamAddMan     $gccDir/man
        _foamAddPath    $gccDir/bin

        # add compiler libraries to run-time environment
        _foamAddLib     $gccDir/lib$WM_COMPILER_LIB_ARCH

        # add gmp/mpfr libraries to run-time environment
        _foamAddLib     $gmpDir/lib
        _foamAddLib     $mpfrDir/lib

        # add mpc libraries (not need for older gcc) to run-time environment
        if [ -n "$mpc_version" ]
        then
            _foamAddLib     $mpcDir/lib
        fi

        # used by boost/CGAL:
        export MPFR_ARCH_PATH=$mpfrDir
        export GMP_ARCH_PATH=$gmpDir
    fi
    unset gcc_version gccDir
    unset gmp_version gmpDir  mpfr_version mpfrDir  mpc_version mpcDir

    if [ -n "$clang_version" ]
    then
        clangDir=$WM_THIRD_PARTY_DIR/platforms/$WM_ARCH$WM_COMPILER_ARCH/$clang_version

        # Check that the compiler directory can be found
        [ -d "$clangDir" ] || {
            echo 1>&2
            echo "Warning in $WM_PROJECT_DIR/etc/config/settings.sh:" 1>&2
            echo "    Cannot find $clangDir installation." 1>&2
            echo "    Please install this compiler version or if you wish to use the system compiler," 1>&2
            echo "    change the 'foamCompiler' setting to 'system'" 1>&2
            echo 1>&2
        }

        _foamAddMan     $clangDir/share/man
        _foamAddPath    $clangDir/bin
    fi
    unset clang_version clangDir
    ;;
system)
    case "$WM_COMPILER" in
    Clang)
        export WM_CC='clang'
        export WM_CXX='clang++'
        clang_version=llvm-9.0.0

        cudaHome="$(dirname $(dirname $(which nvcc)))"
        [ -d "$cudaHome" ] || {
            echo 1>&2
            echo "Error in $WM_PROJECT_DIR/etc/config/settings.sh:" 1>&2
            echo "    Cannot find CUDA installation." 1>&2
            echo "    Found only $cudaHome." 1>&2
            echo 1>&2
            exit 1
        }
        export CUDA_HOME=$cudaHome
        #an option to override thrust
        if [ -d "$THRUST_HOME" ]
        then
            export THRUST_INCLUDE=-I$THRUST_HOME
        else
            export THRUST_INCLUDE=
        fi
        unset cudaHome
        ;;
    esac
    # okay, use system compiler
    ;;
*)
    echo "Warn: foamCompiler='$foamCompiler' is unsupported" 1>&2
    echo "   treating as 'system' instead" 1>&2
    ;;
esac


#
# add c++0x flags for external programs
#
if [ -n "$WM_CXXFLAGS" ]
then
    case "$WM_COMPILER" in
    Gcc*++0x)
        WM_CXXFLAGS="$WM_CXXFLAGS -std=c++0x"
        ;;
    esac
fi

[emankov]

Hello @klausbu,

I don't quite understand how your question relates to HIPIFY tools.

[klausbu]

Well, I hippified the RapidCFD source code but all the settings files that call CUDA to compile the code instead of HIP or whatever ROCm software to be replaced remained unchanged. I need to know what to change to call the c++ compiler and hipcc (HIP) and thrust (RapidCFD uses primarily the trust library). There were also libraries like -lcudart which were not removed from compilation related files - which is hopefully all to be done with them.

[emankov]

First, I'm glad to hear that you successfully hipified your sources. Second, as for building your OpenFOAM stuff, unfortunately, it is out of HIPIFY's scope, sorry.