Marenostrum 5

The modules necessary to compile and run HORSES3D are:

module load intel/2023.2.0 load impi/2021.10.0 load mkl/2023.2.0 load metis/5.1.0 load hdf5/1.10.11
export METIS_DIR=$METIS_ROOT

The Makefile should change line 48 to:

INTEL_HPC_FLAGS= -cpp -Ofast -msse4.2 -axAVX,CORE-AVX2 -assume bscc -D_has_Quad -fp-model=fast=2 -ipo-separate

Then, to compile:

make COMPILER=ifort COMM=PARALLEL WITH_METIS=YES WITH_HDF5=YES MODE=HPC

An example slurm script is included below:

#!/bin/bash
#SBATCH --job-name=jp2
## the name of the qos might change
#SBATCH --qos=gp_resa
#SBATCH --chdir=.
#SBATCH --ntasks=56
#SBATCH --cpus-per-task=4
#SBATCH --ntasks-per-node=28
#SBATCH --time=34:00:00
#SBATCH -e RESULTS/err-naca%j.log
#SBATCH -o RESULTS/out-naca%j.log
 
export OMP_SCHEDULE="guided"
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
export SRUN_CPUS_PER_TASK=${SLURM_CPUS_PER_TASK}
 
module purge
module load intel/2023.2.0
module load impi/2021.10.0
module load mkl/2023.2.0
module load metis/5.1.0
module load hdf5/1.10.11
export METIS_DIR=$METIS_ROOT
 
echo "##########################################################################"
echo "#"
echo "# Running with $SLURM_NTASKS tasks and $SLURM_CPUS_PER_TASK cpus/task"
echo "# On nodes $SLURM_JOB_NODELIST"
echo "#"
echo "##########################################################################"
 
EXEC=../../../bin/horses3d.ns
 
srun $EXEC IA_KG_wr_iles_v15_CFL10.control

To send the jod indicate the account, i.e.:

sbatch -A upm97 -q gp_resa run.sh