Nibbler GPU

Compiler:

**NVFORTRAN **

We need the nvidia SDK and compiler to generate code for the FORTRAN/OpenACC implementation in Horses3D.

List of modules needed to compile and run:

module purge
module load hpc_sdk nvhpc/24.11 gnu14/14.2.0 metis/5.1.0 hdf5/1.14.5-nvhpc24.11-mpi-nof16

export NVHPC_CUDA_HOME=/opt/ohpc/pub/apps/nvidia/hpc_sdk/Linux_x86_64/24.11/cuda
export NVCOMPILER_CUDA_HOME=/opt/ohpc/pub/apps/nvidia/hpc_sdk/Linux_x86_64/24.11/cuda
export NVCOMPILER_COMM_LIBS_HOME=/opt/ohpc/pub/apps/nvidia/hpc_sdk/Linux_x86_64/24.11/comm_libs/12.6
export NVCOMPILER_MATH_LIBS_HOME=/opt/ohpc/pub/apps/nvidia/hpc_sdk/Linux_x86_64/24.11/math_libs/12.6
export OPAL_PREFIX=/opt/ohpc/pub/apps/nvidia/hpc_sdk/Linux_x86_64/24.11/comm_libs/12.6/hpcx/hpcx-2.20/ompi

Then, to compile the NS (compressible Navier-Stokes) solver:

make ns COMPILER=nvfortran COMM=PARALLEL WITH_METIS=YES WITH_HDF5=YES

Slurm scripts

An example slurm script is included below:

#!/bin/bash
#SBATCH --job-name=cylinder
#SBATCH --ntasks=2
#SBATCH --partition=gpu
#SBATCH --gres=gpu:2  # number of gpus max 4 
#SBATCH --mem=32G
#SBATCH --time=00:10:00
#SBATCH --output=RESULTS/slurm-%j.out
#SBATCH --error=RESULTS/slurm-%j.err

module purge
module load hpc_sdk
module load nvhpc/24.11
module load gnu14/14.2.0
module load metis/5.1.0
module load hdf5/1.14.5-nvhpc24.11-mpi-nof16

export NVHPC_CUDA_HOME=/opt/ohpc/pub/apps/nvidia/hpc_sdk/Linux_x86_64/24.11/cuda
export NVCOMPILER_CUDA_HOME=/opt/ohpc/pub/apps/nvidia/hpc_sdk/Linux_x86_64/24.11/cuda
export NVCOMPILER_COMM_LIBS_HOME=/opt/ohpc/pub/apps/nvidia/hpc_sdk/Linux_x86_64/24.11/comm_libs/12.6
export NVCOMPILER_MATH_LIBS_HOME=/opt/ohpc/pub/apps/nvidia/hpc_sdk/Linux_x86_64/24.11/math_libs/12.6
export OPAL_PREFIX=/opt/ohpc/pub/apps/nvidia/hpc_sdk/Linux_x86_64/24.11/comm_libs/12.6/hpcx/hpcx-2.20/ompi
export CUDA_VISIBLE_DEVICES=0

mpirun -np 2 --bind-to none ../horses3d-gpu-main/Solver/bin/horses3d.ns Cylinder.control

To send the job, i.e.:

sbatch run.sh