Using FDS on roihu.csc.fi

These notes provide basic information about using FDS on roihu.csc.fi, of CSC, Finland. It is assumed you have an account and have logged in for the first time.

1. Create ssh keys

   ssh-keygen -t ed25519 -C "youremail@example.com" 
   

and add the public key to your Github ssh keys. See [generating ssh keys](https://help.github.com/articles/generating-ssh-keys/).

2. Clone the [firemodels/fds repo](https://github.com/firemodels/fds) to one of your `/projappl/project_xxx` folders. Use these [notes](https://github.com/firemodels/fds/wiki/Git-Notes-Getting-Started). 

3. In the directory `Build/ompi_gnu_linux`,  run the script to compile  

./make_fds.sh

4. Prepare a SLURM script like this in a folder under your /scratch/project_xxx (in this example, the name of the script is `job_name_script.sh`):

#!/bin/bash #SBATCH --job-name=ductFlow #SBATCH --account=project_xxx #SBATCH --time=05:00:00 #SBATCH --mem-per-cpu=1G #SBATCH --partition=small #SBATCH --ntasks=8 #SBATCH --cpus-per-task=4 #SBATCH --tasks-per-node=8 export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK srun /projappl/project_xxx/firemodels/fds/Build/ompi_gnu_linux/fds_ompi_gnu_linux duct_flow.fds

This script starts 8 tasks (8 mesh) and four OpenMP threads for each task. Roihu has 486 CPU nodes, and 2 x 192 cores per node. All tasks (i.e. 32 threads) can fit into a single node. One GB memory is reserved for each thread, and five hours for the whole thing.

Add the following command to the end of job_name_script.sh to print out the maximum memory requirement during the job:

sacct -o reqmem,maxrss -j $SLURM_JOBID

Another way to collect information about job is to add `seff` command.

5. Run the job by submitting the script:

$ sbatch job_name_script.sh

6. To run the **FDS verification suite**, there are two options:

To run using the queuing system, go to fds/Verification/scripts and run

$ export SLURM_MEMPERCPU=4G $ export SBATCH_ACCOUNT='project_XXX' $ export USE_MAX_CORES=1 $ ./Run_FDS_Cases.sh -q small -w 06:00:00

These commands will first set the memory requirement for computing nodes, and then run both normal and benchmarking   cases on the 'serial' queue. It assumes the jobs can be run in 6 h. To check how close to the 4 GB limit you got, you can list the memory usages between 3 and 4 GB since date YYYY-MM-DD

sacct -o maxrss -S YYYY-MM-DD --units=G | grep '3...G'

Another option is to use interactive shell:

$ sinteractive --account project_XXX --time 48:00:00 --mem 16000 --tmp 100 $ ./Run_FDS_Cases.sh -q none

Once run the cases, create Smokeview images by running

$ ./Make_FDS_Pictures.sh -p ~/bin -X