HAFS online tutorial

Welcome to HAFS online tutorial

This tutorial serves as a guide to learning how to use the Hurricane and Forecast System (HAFS) through a series of practical tests that explore the various capabilities of the model.

This tutorial is based on the latest version of HAFS, and contains information about the prerequisites, installation and building process that are also available in the HAFS User's Guide. The tests can be run in any order, although it is recommended to follow the established order for a better understanding of the model.

Prerequisites and setting environment

The examples in this tutorial are tailored be run on the Mississippi State University's High Performance Computing System (MSU-HPC). The MSU-HPC consist of two components, Orion and Hercules that share a InfiniBand interconnect and two Lustre file systems, "/work/" and "/work2/". In this tutorial, we will use Hercules.

1. MSU account to access to MSU-HPC

To get a MSU-HPC, contact your project's Account Manager to submit an account request. Once the account is fully set up, you can login to MSU-HPC. To login to Orion or Hercules via SSH, you will have to use your MSU account username, MSU password, and Duo two-factor authentication.

2. MSU Hercules login

Hercules login nodes are available externally via SSH.

Linux/Mac users can use the standard ssh client for logging in: ssh -Y username@hercules-login.hpc.msstate.edu

Windows users can use one of the well-known clients such as Putty or Windows PowerShell.
- To install Putty and Xming (for X11 window forwarding) if they are not yet installed/available on your device:
  - Putty: https://www.putty.org/
  - Xming: https://sourceforge.net/projects/xming/
- You can login using Putty as shown below:
- To use Windows PowerShell
  $env:DISPLAY='localhost:0.0'
  ssh -Y username@hercules-login.hpc.msstate.edu

3. Setting up proper environment

After logging into Hercules, the next step is to set up the proper shell environment. To do so, you can edit ~/.bashrc and source ~/.bashrc. This step involves editing files. To become more familiar with text editors and basic commands, refer to the reference section for a Vim cheatsheet. It can also be useful to acquaint yourself with basic Linux commands, see the refenrence section for a Linux commands cheatsheet.

Edit .bashrc and add the following:

$ vi ~/.bashrc

module use /apps/contrib/modulefiles
module load rocoto
module load noaatools
module use /work/noaa/hwrf/soft/modulefiles
module load emc-utils/1.1.0
module load imagemagick

Optional to alias for slurm related commands in .bashrc

alias sq='squeue'
alias squ='squeue -u ${USER}'
alias sqj='squeue -j'
alias sqp='squeue -p'
alias sqq='squeue -q'
alias si='sinfo'
alias ssj='scontrol show job'

Also optional to alias for commonly used dirs in .bashrc

alias tsv2='cd /work2/noaa/hwrf/tutorial/save/${USER}'
alias tns2='cd /work2/noaa/hwrf/tutorial/noscrub/${USER}'
alias tsc2='cd /work2/noaa/hwrf/tutorial/scrub/${USER}'

Then source .bashrc

$ source ~/.bashrc

Downloading, building and installing HAFS

This section provides the instructions to obtain the source code from the HAFS Github repository, build and install HAFS, and configure the model to run it. The guidelines provided here to download, build and install HAFS are customized for this tutorial. For more detailed directions on downloading any other branch from the HAFS Github repository, please refer to the HAFS User's Guide

1. Creating directory

Before cloning the HAFS repository, you will need to create a directory to copy the source code and run HAFS, tokeep the tutorial tests separte from other work.

$ mkdir -p /path-of-hafs-directory/
$ cd /path-of-hafs-directory/

2. Cloning the HAFS Github repository

Once you are in the directory where you want to copy the source code,the next step is to clone the source code from the HAFS Github repository using the following command:

$ git clone <-b BRANCH> --recursive https://github.com/hafs-community/HAFS.git ./

The BRANCH for this tutorial is feature/hafs-ncas-m
Simply set <-b BRANCH> option to the branch name

3. Building and installing HAFS

After cloning and checking out the tutorial branch, go to the sorc directory to build and install HAFS

$ cd /path-of-hafs-directory/sorc
$ ./install_hafs.sh > install_hafs.log 2>&1

This step could take ~30 mins or longer. If you got any errors during this step, look into the '/sorc/logs' directory

The install_hafs.sh script conducts the following steps:

Run ./build_all.sh to build executables for all HAFS subcomponents and tools

Run build_forecast.sh, build_post.sh, build_tracker.sh, build_utils.sh, build_tools.sh, build_gsi.sh, build_hycom_utils.sh, build_ww3_utils.sh

The above build_*.sh can also run individually to build the specific subcomponent only

Run ./install_all.sh to copy and install all executables

Run ./link_fix.sh to link fix directories and files

Copy platform specific system.conf file

4. Configuring and running HAFS

After succesfully installing HAFS, check and edit HAFS system.conf, and ensure that it has the appropiate configuration

$ cd /path-of-hafs-directory/parm
$ cp system.conf.<'''system'''> system.conf
$ vi system.conf

where <'''system'''> in this case is hercules

disk_project=hwrf
tape_project=emc-hwrf
cpu_account=hurricane
archive=disk:/work2/noaa/{disk_project}/tutorial/noscrub/{ENV[USER]}/hafsarch/{SUBEXPT}/{out_prefix}.tar
CDSAVE=/work2/noaa/{disk_project}/tutorial/save/{ENV[USER]}
CDNOSCRUB=/work2/noaa/{disk_project}/tutorial/noscrub/{ENV[USER]}/hafstrak
CDSCRUB=/work2/noaa/{disk_project}/tutorial/scrub/{ENV[USER]}
syndat=/work/noaa/hwrf/noscrub/input/SYNDAT-PLUS

To run HAFS, change directories and go to the /rocoto directory to modify the workflow driver (e.g. cronjob_hafs.sh). Make sure HOMEhafs is correct

$ cd /path-of-hafs-directory/rocoto
$ vi cronjob_hafs.sh

After modifying the script, you can repeat running the driver periodically, or add it as a cron task to advance the workflow

Manually:
$ ./cronjob_hafs.sh
Set up a cron task to run periodically:
$ crontab -e

*/5 * * * * /path-of-hafs-directory/rocoto/cronjob_hafs.sh

To set up the cron task you need to login login-node-1. This can be done with the following command:
$ ssh hercules-login-1

The practical tests in this tutorial will guide you to further explore and comprehend the steps to run HAFS with specific configurations.

Online Tutorial Tests

Test case (Hurricane Ida 2021 09L)

Hurricane Ida made landfall near Port Fourchon, Louisiana, on August 29, 2021, as a Category 4 hurricane with sustained winds of 150 mph. Ida is one of the costliest Atlantic hurricanes in the United States, having caused at least $75.25 billion (2021 USD) in damages. A total of 107 deaths were attributed to Ida, including 87 in the USA and 20 in Venezuela.

alt text

Test 0: Regional standalone 6km resolution storm-focused configuration

This first test will generate a 12h forecast for the 20210827 12Z cycle of Hurricane Ida (2021), using a regional standalone 6-km resolution storm-focused configuration with the Extended Schmidt Gnomonic (ESG grid), with no moving nest. The run uses GFS (Global Forecast System) 0.25 degree resolution grib2 format input data for atmospheric initial and lateral boundary conditions. The test is a coldstart run from GFS. We won't be using vortex initialization or data assimilation. The test uses an atmospheric only configuration, with no ocean coupling and uses the GFDL MP scheme.

Checking the cronjob driver

To run HAFS with the configuration described above we are going to use the cronjob driver cronjob_hafs_tutorial_test0.sh. To do so you will first have to change into the rocoto directory. where the cronjob driver is located, and check that the script has the right information, making sure that the HOMEhafs dir is correct.

$ cd /path-of-hafs-directory/rocoto
$ vi cronjob_hafs_tutorial_test0.sh

alt text

As shown in the example above, the cronjob driver establishes the cycle and storm to be run and configuration

Reviewing the configuration

Before running the cronjob driver, we are going to review the configuration of the run, to make sure it matches the intended configuration. To do so, we change into the parm directory and check the configuration file defined in the cronjob driver hafs_regional_atm.conf

Running the cronjob driver

After reviewing the configuration, it is time to run the driver. To do so change back into the rocoto directory and execute the driver.

Manually:
$ ./cronjob_hafs_tutorial_test0.sh
Set up a cron task to run periodically:
On Hercules, cron is only available on the hercules-login-1 node
$ ssh hercules-login-1
$ crontab -e

*/4 * * * * /bin/sh -l -c /path-of-hafs-directory/rocoto/cronjob_hafs_tutorial_test0.sh >> /path-of-hafs-directory/rocoto/cronjob_hafs_tutorial_test0.log 2>&1

To check the rocoto workflow related files, after running the driver, use the following command to list the files

$ ls -1 hafs-hafs_202405_tutorial_test0-09L-2021082712*

The following files should be generated:

alt text

The cronjob driver should also have submitted the launch or other jobs/tasks.

Checking and monitoring the Rocoto workflow

You can check the Slurm jobs through "squ"

$ squeue -u ${USER}

Or through the "qac" command

$ qac

Note: if qac command does not work, please run the “gem install rexml” command first.

You can use Rocoto commands to check the status and rewind the Rocoto worflow taks. For example:

$ rocotostat -w hafs-hafs_202405_tutorial_test0-09L-2021082712.xml -d hafs-hafs_202405_tutorial_test0-09L-2021082712.db

alt text

For more information about rocoto:
https://github.com/christopherwharrop/rocoto/
https://christopherwharrop.github.io/rocoto/

You can also check the status of the rocoto workflow and rewind failed tasks with rocoto/rocoto_util.sh

You can use the script for a specific workflow as follows:

Check active tasks for a specific workflow:
- $ ./rocoto_util.sh -a hafs-hafs_202405_tutorial_test0-09L-2021082712

Check task status for a specific workflow:
- $ ./rocoto_util.sh -s hafs-hafs_202405_tutorial_test0-09L-2021082712

Rewind failed tasks for a specific workflow:
- $ ./rocoto_util.sh -r hafs-hafs_202405_tutorial_test0-09L-2021082712

Alternatively, you can check active tasks, all task status and rewind all failed tasks for all workflows under current directory as follows:

$ ./rocoto_util.sh -a

$ ./rocoto_util.sh -s

$ ./rocoto_util.sh -r

Checking workflow running directories and job logs

Check Job logs

When running HAFS, it might be necessary to check the jobs log files. Typically the log files are located in the WORKhafs directory with the following naming convention:

WORKhafs=/CDSCRUB_directory/experiment_name/cycle_date_and_time/storm_id

To find the directory:
$ cat /path-of-hafs-directory/parm/system.conf

Search for CDSCRUB and list files

$ ls /CDSCRUB_directory/experiment_name/cycle_date_and_time/storm_id

For example, in Hercules when running experiment hafs_202405_tutorial_test0 for Ida (09L) for cycle 2021082712 in directory:
HOMEhafs=/work2/noaa/hwrf/tutorial/save/${USER}/hafs_202405

The log files are located in the directory:
WORKhafs=/work2/noaa/hwrf/tutorial/scrub/${USER}/hafs_202405_tutorial_test0/2021082712/09L

To list the log files:
$ ls /work2/noaa/hwrf/tutorial/scrub/${USER}/hafs_202405_tutorial_test0/2021082712/09L

alt text

Check workflow products under COMhafs

The workflow products are located under the COMhafs directory with the following naming convention:
COMhafs=/CDSCRUB_directory/com/experiment_name/cycle_date_and_time/storm_id

To find the directory:
$ cat /path-of-hafs-directory/parm/system.conf

Search for CDSCRUB

$ ls /CDSCRUB_directory/experiment_name/cycle_date_and_time/storm_id

For example, in Hercules when running experiment hafs_202405_tutorial_test0 for Ida (09L) for cycle 2021082712 in directory:
HOMEhafs=/work2/noaa/hwrf/tutorial/save/${USER}/hafs_202405

The workflow products are located in the directory:
COMhafs=/work2/noaa/hwrf/tutorial/scrub/${USER}/hafs_202405_tutorial_test0/com/2021082712/09L

$ ls /work2/noaa/hwrf/tutorial/scrub/${USER}/hafs_202405_tutorial_test0/com/2021082712/09L

alt text

Viewing results using NCVIEW

After succesfully running HAFS, check forecast job log and results use ncview to view forecast history outputs (atmfhhh.nc,sfcfhhh.nc, e.g.)

Load the necessary modules:
$ module use /path-of-hafs-directory/modulefiles
$ module load hafs.hercules
$ module load ncview
$ module list

Change into the directory with the results

$ cd /CDSCRUB_directory/experiment_name/cycle_date_and_time/storm_id/forecast

To check the forecast job log, you can use the following command:
$ vi forecast.log

To use ncview to view forecast output you can use the following command: $ ncview NETCDF_FORMAT_FILE

For example:
$ ncview atmf012.nc &
$ ncview sfcf012.nc &

This will show the 12-h forecast results

Test 1: Regional storm-focused moving-nesting configuration

The test described in this section is similar to Test 0 that employed a regional standalone 6-km resolution storm-focused configuration, but this time using moving-nesting. This test will generate a 12h forecast for the 20210827 12Z cycle of Hurricane Ida (2021), with a regional coarse-resolution storm-focused moving-nesting configuration with the Extended Schmidt Gnomonic (ESG grid). This test uses a 12-km resolution parent domain with a 4-km resolution storm-following moving nest. For initial conditions (IC) we will use GFS NetCDF format analysis, and 0.25 degree resolution grib2 formast forecast for lateral boundary conditions (LBC) The test is a coldstart run from GFS. We won't be using vortex initialization or data assimilation. The test uses an atmospheric only configuration, with no ocean coupling and uses a HAFSv2A-like NATL physics suite (with Thompson microphysics). As shown in the diagram below, the workflow of this test is similar to the workflow for Test 0, with the addition of the atm_prep_mvnest task.