Magerit GPU

Compiler:

**NVFORTRAN ** (last checked:01/06/2025)

We need the nvidia SDK and compiler to generate code for the FORTRAN/OpenACC implementation in Horses3D.

List of modules needed to compile and run:

module purge
module load NVHPC/23.7-CUDA-12.2.0
export LD_LIBRARY_PATH=//media/apps/avx512-2021/software/NVHPC/23.7-CUDA-12.2.0/Linux_x86_64/23.7/comm_libs/mpi/lib/:$LD_LIBRARY_PATH
export PATH=/media/apps/avx512-2021/software/NVHPC/23.7-CUDA-12.2.0/Linux_x86_64/23.7/comm_libs/mpi/bin:$PATH

Then, to compile the NS (compressible Navier-Stokes) solver:

make ns COMPILER=nvfortran ENABLE_THREADS=NO COMM=PARALLEL

or to compile the MU (multiphase) solver:

make mu COMPILER=nvfortran ENABLE_THREADS=NO COMM=PARALLEL

If a signle GPU run is to be used then the last one options can be ommited. For parallel cases, use partitioning=SFC in the control file.

Slurm scripts

An example slurm script is included below:

#!/bin/bash
##----------------------- Start job description -----------------------
#SBATCH --job-name=testGpu
#SBATCH --ntasks=2
#SBATCH --partition=standard-gpu
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=20G
#SBATCH --gres=gpu:a100:2
#SBATCH --time=00:10:00
#SBATCH -e RESULTS/err-test%j.log
#SBATCH -o RESULTS/out-test%j.log

module purge
module load NVHPC/23.7-CUDA-12.2.0
export LD_LIBRARY_PATH=//media/apps/avx512-2021/software/NVHPC/23.7-CUDA-12.2.0/Linux_x86_64/23.7/comm_libs/mpi/lib/:$LD_LIBRARY_PATH
export PATH=/media/apps/avx512-2021/software/NVHPC/23.7-CUDA-12.2.0/Linux_x86_64/23.7/comm_libs/mpi/bin:$PATH

echo "##########################################################################"
echo "#"
echo "# Running with $SLURM_NTASKS tasks and $SLURM_CPUS_PER_TASK cpus/task"
echo "# On nodes $SLURM_JOB_NODELIST"
echo "#"
echo "##########################################################################"

EXEC= PATH_T0_HORSES_EXECUTABLE 

# For parallel runs
mpirun $EXEC caseFile.control

# For serial runs
#srun --unbuffered $EXEC caseFile.control

To send the job, i.e.:

sbatch run.sh