Skip to content

Run setup scripts to generate experiment

Jump to bottom
Kate Friedman edited this page Feb 8, 2022 · 29 revisions

To run the setup scripts, you need to make sure to have a copy of python3 with numpy available. The easiest way to guarantee this is to load python from the official hpc-stack installation for the machine you are on:

Hera:

module use -a /contrib/anaconda/modulefiles
module load anaconda/anaconda3-5.3.1

Orion:

module load python/3.7.5

WCOSS-Dell:

module load python/3.6.3

WCOSS2:

module load python/3.8.6

If running with Rocoto make sure to have a Rocoto module loaded before running setup scripts:

Hera:

module load rocoto/1.3.3

Orion:

module load contrib
module load rocoto/1.3.3

WCOSS-Dell:

module use /usrx/local/dev/emc_rocoto/modulefiles
module load rocoto/1.3.0rc2

Free-forecast experiment

Scripts that will be used:

  • ush/rocoto/setup_expt.py
  • ush/rocoto/setup_workflow_fcstonly.py

1) Run experiment generator script (creates EXPDIR and COMROT)

NOTE: The following command examples include variables for reference but users should not use environmental variables to submit the commands. Exporting variables like EXPDIR to your environment causes an error when the python scripts run. Please explicitly include the argument inputs when running both setup scripts.

cd ush/rocoto
./setup_expt.py forecast-only --app $APP --pslot $PSLOT --configdir $CONFIGDIR --idate $IDATE --edate $EDATE --resdet $RES --gfs_cyc $GFS_CYC --comrot $COMROT --expdir $EXPDIR

...where:

  • forecast-only is the first positional argument that instructs the setup script to produce an experiment directory for forecast only experiments.
  • $APP is the target application, one of:
    • ATM: atmosphere-only [default]
    • ATMW: atm-wave
    • S2S: atm-ocean-ice
    • S2SW: atm-ocean-ice-wave
  • $PSLOT is the name of your experiment
  • $CONFIGDIR is the path to the /config folder under the copy of the system you're using (i.e. $PATH_TO_CLONE/parm/config/)
  • $IDATE is the initial start date of your run (first cycle CDATE, YYYYMMDDCC)
  • $EDATE is the ending date of your run (YYYYMMDDCC) and is the last cycle that will complete
  • $RES is the FV3 resolution (i.e. 768 for C768)
  • $GFS_CYC is the forecast frequency (0 = none, 1 = 00z only [default], 2 = 00z & 12z, 4 = all cycles)
  • $COMROT is the path to your experiment output directory. DO NOT include PSLOT folder at end of path, it’ll be built for you.
  • $EXPDIR is the path to your experiment directory where your configs will be placed and where you will find your workflow monitoring files (i.e. rocoto database and xml file). DO NOT include PSLOT folder at end of path, it will be built for you.

Examples:

Atm-only

cd ush/rocoto
./setup_expt.py forecast-only --pslot test --idate 2020010100 --edate 2020010118 --resdet 384 --gfs_cyc 4 --comrot /some_large_disk_area/Joe.Schmo/comrot --expdir /some_safe_disk_area/Joe.Schmo/expdir

Coupled

cd ush/rocoto
./setup_expt.py forecast-only --app S2SW --pslot coupled_test --idate 2013040100 --edate 2013040100 --resdet 384 --comrot /some_large_disk_area/Joe.Schmo/comrot --expdir /some_safe_disk_area/Joe.Schmo/expdir --icsdir /some_large_disk_area/Joe.Schmo/icsdir

2) Set user and experiment settings

Go to your EXPDIR and check/change the following variables within your config.base now before running the next script:

  • ACCOUNT
  • HOMEDIR
  • STMP
  • PTMP
  • ARCDIR (location on disk for online archive used by verification system)
  • HPSSARCH (YES turns on archival)
  • HPSS_PROJECT (project on HPSS if archiving)
  • ATARDIR (location on HPSS if archiving)

Some of those variables will be found within a machine-specific if-block so make sure to change the correct ones for the machine you'll be running on.

Now is also the time to change any other variables/settings you wish to change in config.base or other configs. Do that now. Once done making changes to the configs in your EXPDIR go back to your clone to run the second setup script.

See the following section of this wiki for more information on configuring your run:

https://github.com/NOAA-EMC/global-workflow/wiki/Configure-your-run

3) Run workflow generator script (creates ROCOTO xml in EXPDIR)

./setup_workflow_fcstonly.py --expdir $EXPDIR/$PSLOT

Example:

./setup_workflow_fcstonly.py --expdir /some_safe_disk_area/Joe.Schmo/expdir/test

4) Resulting files from setup scripts

You will now have a rocoto xml file in your EXPDIR ($PSLOT.xml) and a crontab file generated for your use. If you do not have a crontab file you may not have had the rocoto module loaded. To fix this: load a rocoto module and then rerun setup_workflow*.py script again. Cron is handled differently on WCOSS-Dell so follow different instructions for setting up your rocoto cron on Mars/Venus.

Cycled experiment

Scripts that will be used:

  • ush/rocoto/setup_expt.py
  • ush/rocoto/setup_workflow.py

1) Run experiment generator script (creates EXPDIR and COMROT)

NOTE: The following command examples include variables for reference but users should not use environmental variables to submit the commands. Exporting variables like EXPDIR to your environment causes an error when the python scripts run. Please explicitly include the argument inputs when running both setup scripts.

cd ush/rocoto
./setup_expt.py cycled --pslot $PSLOT --configdir $CONFIGDIR --idate $IDATE --edate $EDATE --comrot $COMROT --expdir $EXPDIR [ --icsdir $ICSDIR --resdet $RESDET --resens $RESENS --nens $NENS --gfs_cyc $GFS_CYC ]

(Coupling cycled not yet supported)

Example:

cd ush/rocoto
./setup_expt.py cycled --pslot test --configdir /home/Joe.Schmo/git/global-workflow/parm/config --idate 2020010100 --edate 2020010118 --comrot /some_large_disk_area/Joe.Schmo/comrot --expdir /some_safe_disk_area/Joe.Schmo/expdir --resdet 384 --resens 192 --nens 80 --gfs_cyc 4

...where:

  • cycled is the first positional argument that instructs the setup script to produce an experiment directory for cycled experiments.
  • $PSLOT is the name of your experiment
  • $CONFIGDIR is the path to the /config folder under the copy of the system you're using (i.e. $PATH_TO_CLONE/parm/config/)
  • $IDATE is the initial start date of your run (first cycle CDATE, YYYYMMDDCC)
  • $EDATE is the ending date of your run (YYYYMMDDCC) and is the last cycle that will complete
  • $ICSDIR is the path to the ICs for your run if generated separately.
  • $COMROT is the path to your experiment output directory. Do not use noscrub space on Cray for COMROT, use ptmp. DO NOT include PSLOT folder at end of path, it’ll be built for you.
  • $EXPDIR is the path to your experiment directory where your configs will be placed and where you will find your workflow monitoring files (i.e. rocoto database and xml file). DO NOT include PSLOT folder at end of path, it will be built for you.
  • $RESDET is the FV3 resolution of the deterministic forecast (i.e. ‘--resdet 768’, optional, default is C384)
  • $RESENS is the FV3 resolution of the ensemble (EnKF) forecast (i.e. ‘--resens 384’, optional, default is C192)
  • $NENS is the number of ensemble members (optional, default is 20)
  • $GFS_CYC is the cycle frequency of the long GFS forecast (0 = none, 1 = 00z only [default], 2 = 00z & 12z, 4 = all cycles)

Example setup_expt.py on WCOSS_C:

SURGE-slogin1 > ./setup_expt.py cycled --pslot fv3demo --idate 2017073118 --edate 2017080106 --comrot /gpfs/hps2/ptmp/Joe.Schmo --expdir /gpfs/hps3/emc/global/noscrub/Joe.Schmo/para_gfs

SDATE = 2017-07-31 18:00:00
EDATE = 2017-08-01 06:00:00

EDITED:  /gpfs/hps3/emc/global/noscrub/Joe.Schmo/para_gfs/fv3demo/config.base as per user input.
DEFAULT: /gpfs/hps3/emc/global/noscrub/Joe.Schmo/para_gfs/fv3demo/config.base.default is for reference only.
Please verify and delete the default file before proceeding.

SURGE-slogin1 >

The message about the config.base.default is telling you that you are free to delete it if you wish but it’s not necessary to remove. Your resulting config.base was generated from config.base.default and the default one is there for your information.

What happens if I run setup_expt.py again for an experiment that already exists:

SURGE-slogin1 > ./setup_expt.py forecast-only --pslot fv3demo --idate 2017073118 
--edate 2017080106 --comrot /gpfs/hps2/ptmp/Joe.Schmo --expdir /gpfs/hps3/emc/global/noscrub/Joe.Schmo/para_gfs

COMROT already exists in /gpfs/hps2/ptmp/Joe.Schmo/fv3demo

Do you wish to over-write COMROT [y/N]: y

EXPDIR already exists in /gpfs/hps3/emc/global/noscrub/Joe.Schmo/para_gfs/fv3demo

Do you wish to over-write EXPDIR [y/N]: y

SDATE = 2017-07-31 18:00:00
EDATE = 2017-08-01 06:00:00

EDITED:  /gpfs/hps3/emc/global/noscrub/Joe.Schmo/para_gfs/fv3demo/config.base as per user input.
DEFAULT: /gpfs/hps3/emc/global/noscrub/Joe.Schmo/para_gfs/fv3demo/config.base.default is for reference only.
Please verify and delete the default file before proceeding.

Your COMROT and EXPDIR will be deleted and remade. Be careful with this!

2) Set user and experiment settings

Go to your EXPDIR and check/change the following variables within your config.base now before running the next script:

  • ACCOUNT
  • HOMEDIR
  • STMP
  • PTMP
  • ARCDIR (location on disk for online archive used by verification system)
  • HPSSARCH (YES turns on archival)
  • HPSS_PROJECT (project on HPSS if archiving)
  • ATARDIR (location on HPSS if archiving)

Some of those variables will be found within a machine-specific if-block so make sure to change the correct ones for the machine you'll be running on.

Now is also the time to change any other variables/settings you wish to change in config.base or other configs. Do that now. Once done making changes to the configs in your EXPDIR go back to your clone to run the second setup script.

See the following section of this wiki for more information on configuring your run:

https://github.com/NOAA-EMC/global-workflow/wiki/Configure-your-run

3) Run workflow generator script (creates ROCOTO xml in EXPDIR)

./setup_workflow.py --expdir $EXPDIR/$PSLOT

Example:

./setup_workflow.py --expdir /some_safe_disk_area/Joe.Schmo/expdir/test

4) Resulting files from setup scripts

You will now have a rocoto xml file in your EXPDIR ($PSLOT.xml) and a crontab file generated for your use. If you do not have a crontab file you may not have had the rocoto module loaded. To fix this: load a rocoto module and then rerun setup_workflow*.py script again. Cron is handled differently on WCOSS-Dell so follow different instructions for setting up your rocoto cron on Mars/Venus.

Users running GFSv16

Clone this wiki locally