Skip to content

WRF Chem Upgrade to Manhattan

Jump to bottom
Tim Hoar edited this page Nov 10, 2020 · 13 revisions

Procedures to Build & Run (NCAR’s Cheyenne)

  1. Obtaining the codes
  2. Build the Required Code (#2-build all in same top-level directory location
  3. Get Test data from Globus
  4. Run DART/WRF-Chem Tutorial
  5. Check results

1. Obtain Required Code and Datasets for Tutorial

ED Note: The fossell/DART_development final_wrfchem_upgrade branch was merged onto the NCAR/DART_development chem_upgrade branch with PR #66

Checkout DART/WRF-Chem branch "final_wrfchem_upgrade" from Github repo https://github.com/fossell/DART_development

Checkout DART/WRF-Chem branch "chem_upgrade" from Github repo https://github.com/NCAR/DART_development

Checkout "chem_upgrade" branch from Arthur's Github repository. (you will need to contact Arthur and provide your GitHub username to be given permission to read his repository) https://github.com/apmizzi/DART_development

>> git clone https://github.com/apmizzi/DART_development  dart_wrfchem
>> cd dart_wrfchem
>> git checkout chem_upgrade
  • Note: $DART_DIR refers to top of DART code tree, e.g. dart_wrfchem/

2. Build the Required Code (build all in same top-level directory location)

Build WRF Chem (v3.9.1)

  • Helpful information links:

  • Unpack WRFV3.9.1

    >>  cd WRFV3
    • Unpack WRF-Chem v3.9.1 inside WRFV3 [Creates subdirectory WRFV3/chem]

  • Set the proper environment variables

    >> setenv EM_CORE  1
>> setenv NMM_CORE 0
>> setenv WRF_CHEM 1
>> setenv WRF_KPP 0
>> setenv YACC “/usr/bin/yacc -d”
>> setenv FLEX_LIB_DIR /usr/lib64

  • Load the proper modules:

    1) ncarenv/1.3   2) intel/18.0.5   3) ncarcompilers/0.5.0
4) mpt/2.19      5) netcdf/4.6.3   6) xxdiff/4.0.1
7) mkl/2018.0.5  8) nco/4.7.9      9) ncl/6.6.2

  • Configure and Compile

    >> ./configure
     [Option 24 (intel, dmpar, sgi mpt)]
>> ./compile  em_real  >&  compile.log
     [Look for executables for successful build]

Build WRFDA

See WRFDA documentation for assistance

Build WPS

See WRF/WPS documentation for assistance

WPS_GEOG (WPS Static dataset)

  • Link from “wrfhelp” on Cheyenne to location where all of the codes are built. 
    >> ln  -s  /glade/u/home/wrfhelp/WPS_GEOG/  GEOG_DATA
[Tutorial script by default expects directory name 'GEOG_DATA', but can change this in the script if you want.]

Build DART/wrf_chem

  • Make sure MKL libraries are loaded [module load mkl]

    • Set up build environment (NCAR Cheyenne example)
    >> cd $DART_DIR/build_templates
>> cp  mkmf.template.intel.linux  mkmf.template
    • Build the code
    >> cd $DART_DIR/models/wrf_chem/work
>> ./quickbuild.csh  >& build.log &
    • Build the observation converters
    >> cd $DART_DIR/observations/obs_converters
>> ./buildall.csh  >&  build.log &
    • Build WRF-Chem utilities
    >> cd  $DART_DIR/models/wrf_chem/run_scripts/
>> ./buildall.csh  >&  build.log &

3. Get Test data from Globus

  • Login to https://www.globus.org/
  • Search for Endpoint “WRF/CHEM test data files”
  • Download data files (includes test input data for the tutorial, and sample output data for the tutorial to compare results)

NOTE: the Globus endpoint is purged every 45 days. If you happen to be unlucky enough to find that the Endpoint is empty, please send email to dart@ucar.edu reminding us to restore the purged files.

4. Run DART/WRF-Chem Tutorial

  • Modify System Job Scripts

    >>  cd  $DART_DIR/models/wrf_chem/hybrid_scripts
[Edit] da_run_hold*.ksh

  • Modify Configuration and Runtime settings in top-level script

    >>  cd  $DART_DIR/models/wrf_chem/run_scripts/RUN_REAL_TIME/FINAL_TEST_SCRIPTS

  • Edit real_time_PANDA_RETR_RELEASE_TEST_NCAR_MANHATTAN.ksh

    Important path environment variables to set in run script:

    # CODE VERSIONS (directory name only, not path):
WPS_VER = Name of build WPS code directory
WPS_GEOG_VER= Name of GEOG_DATA directory
WRFDA_VER=Name of build WRFDA code directory
WRF_VER=Name of build WRF code directory
WRFCHEM_VER=Name of build WRF-Chem code directory
DART_VER=Name of build DART code directory

# ROOT DIRECTORIES:
SCRATCH_DIR = directory for running tutorial
WORK_DIR = top level directory where all of the source codes reside
INPUT_DATA_DIR= directory where input data for tutorial was placed and unpacked.

Important settings for initial cycle (2014072418) in script:

INITIAL_DATE=2014072418
FIRST_FILTER_DATE=2014072500
FIRST_DART_INFLATE_DATE=2014072500
FIRST_EMISS_INV_DATE=2014072500

# START CYCLE DATE-TIME:
CYCLE_STR_DATE=2014072418

# END CYCLE DATE-TIME:
CYCLE_END_DATE=2014072418

  • Run script for initial cycle (2014072418)

    >> cd models/wrf_chem/run_scripts/RUN_REAL/FINAL_TEST_SCRIPTS
>> ./real_time_PANDA_RETR_RELEASE_TEST_NCAR_MANHATTAN.ksh >& chemtut_initial.out &

  • Modify and run the script for the first assimilation time 2014072500 Edit the script for new date and cycle settings:

    # START CYCLE DATE-TIME:
CYCLE_STR_DATE=2014072500

# END CYCLE DATE-TIME:
CYCLE_END_DATE=2014072500
RUN_GEOGRID=false
[No need to run geogrid more than once during initial cycle]

>> ./real_time_PANDA_RETR_RELEASE_TEST_NCAR_MANHATTAN.ksh >& chemtut_firstassim.out &

    [Advanced use: the “SELECT COMPONENT RUN OPTIONS” section of the run script may also be of interest after running the tutorial to turn on/off different obs types. Note that doing so may require edits to other parts of the run script, e.g. namelist sections]

5. Check results

  • Located in $SCRATCH_DIR/real_PANDA_RETR_VARLOC_RELEASE_TEST

Summary of Changes

Since the file layout of the SVN wrf-chem.r13172 tag is very different than the layout under Manhattan, it is difficult to determine what files have meaningfully changed, which ones are new, which ones are renamed, etc. Here is a list of files that have changed between wrf-chem.r13172 (Lanai) and final_wrfchem_upgrade (Manhattan).

WRF-Chem Model:

Directory wrf_chem was initiated by making a copy of the Manhattan wrf directory. Files/directories found only in the Lanai-WRF-Chem were copied into the new wrf_chem. Changes to existing files were ported into Manhattan-WRF base versions (e.g. adding chem fields to model_mod.f90). Modifications were made where necessary to make things Manhattan-compliant(e.g. directory structures for paths and mkmf’s).

models/wrf_chem

Filename Version / Mods / Notes
Distrib_phb_mod.f90 Manhattan-WRF version
Distributed_phb_model_mod.f90 Manhattan-WRF version
model_mod.f90 Added WRF-Chem content to Manhattan-WRF version
model_mod.html Manhattan-WRF version
model_mod.nml Manhattan-WRF version
model_mod_vorticity.f90 Manhattan-WRF version
module_map_utils.f90 Manhattan-WRF version
module_map_utils-wrf3.0.f90 Manhattan-WRF version
README Lanai-WRF-Chem version
README_WRF-Chem_DART_BUILD_RUN_INSTRUCTIONS Lanai-WRF-Chem version w/ modifications for new procedures and typos
select.f90 Manhattan-WRF version
stitch_post.sh Manhattan-WRF version
stitch_prior.sh Manhattan-WRF version
variable_localization.tx New feature added to Manhattan-wrf_chem version
wrf_state_variables_table Manhattan-WRF version
Directory Version / Mods / Notes
hybrid_scripts Lanai-WRF-Chem version w/ mods: modify namelist scripts to use correct Manhattan path structure (airnow, iasi, mopitt, ncep_ascii, panda)
namelist_scripts/DART Lanai-WRF-Chem version w/ mods: modify/add to be Manhattan compliant; add obs_impact_tool_nml, quality_control_nml, state_vector_io_nml; modify ensemble_manager, model_nml,perfect_model_obs_nml, filter_nml
namelist Manhattan-WRF version; not used/tested
regression Manhattan-WRF version; not used/tested
experiments Manhattan-WRF version; not used/tested
matlab Lanai-WRF-Chem version; not used/tested
run_diagnostics Lanai-WRF-Chem version; not used/tested
run_diagnostics_OMI Lanai-WRF-Chem version; not used/tested
run_frappe_diagnostics Lanai-WRF-Chem version; not used/tested
run_matlab_scripts Lanai-WRF-Chem version; not used/tested
shell_scripts Lanai-WRF-Chem version; not used/tested
WRF_BC Manhattan-WRF version; not used/tested
WRFCHEM_BUG_FIXES Lanai-WRF-Chem version; not used/tested
PERTURB Manhattan-WRF version; not used/tested
WRF_DART_utilities Manhattan-WRF versions w/ mods: Add chem fields to wrf_dart_obs_preprocess; added extra source files from Lanai-WRF-Chem but aren’t used. (e.g. wrf_chem_dart_obs_preprocess not used, the regular wrf version is used/built in the work dir); wrf_to_dart and dart_to_wrf were not copied over, but do have chem mods for transforms although not used in tutorial.
run_scripts*
* See info an end of section titled “RUN_SCRIPT MODS” for more detail regarding scripts		 Lanai-WRF-Chem version w/ mods: Subdirectory RUN_REAL/FINAL_TEST_SCRIPTS houses the top level script for tutorial RUN_REAL/* contains various source codes and exes (when built) that system use. Tutorial may use: RUN_EMISS_INV, RUN_WES_COLDENS, RUN_MEGAN_BIO, RUN_FINN_FIRE, RUN_PERT_CHEM, RUN_BAND_DEPTH, RUN_TIME_INTERP
work Control_impact_runtime.table New feature added to Manhattan-wrf_chem version
mkmf_* Added new file for chem tools (obs_impact_tools, compar_states). Modified paths to be Manhattan compliant
paths_* Added new file for chem tools (obs_impact_tools, compar_states) Modified paths to be Manhattan compliant
input.nml Lanai-WRF-Chem version w/ mods: Added/modified sections and entries to be Manhattan compliant

observations/forward_operators

File Mods / Notes
obs_def_IASI_CO_mod.f90 New file, Manhattan compliant (state_handle, ens_size, etc), modified mkmf and path files to reflect Manhattan structure
obs_def_IASI_O3_partition_mod.f90 New file, Manhattan compliant (state_handle, ens_size, etc), modified mkmf and path files to reflect Manhattan structure
obs_def_IASI_O3_mod.f90 New file, Manhattan compliant (state_handle, ens_size, etc), modified mkmf and path files to reflect Manhattan structure
obs_def_PANDA_OBS_mod.f90 New file, Manhattan compliant (state_handle, ens_size, etc), modified mkmf and path files to reflect Manhattan structure
obs_def_AIRNOW_OBS_mod.f90 New file, Manhattan compliant (state_handle, ens_size, etc), modified mkmf and path files to reflect Manhattan structure
obs_def_OMI_NO2_mod.f90 New file, Manhattan compliant (state_handle, ens_size, etc), modified mkmf and path files to reflect Manhattan structure
obs_def_MOPITT_CO_mod.f90 New file, Manhattan compliant (state_handle, ens_size, etc), modified mkmf and path files to reflect Manhattan structure
Obs_def_tec_mod.f90 New file, Manhattan compliant (state_handle, ens_size, etc), modified mkmf and path files to reflect Manhattan structure
Obs_def_monitor_mod.f90 New file, Manhattan compliant (state_handle, ens_size, etc), modified mkmf and path files to reflect Manhattan structure
obs_def_MODIS_AOD_mod.f90 New file, Manhattan compliant (state_handle, ens_size, etc), modified mkmf and path files to reflect Manhattan structure
obs_def_gps_mod.f90 Modified Manhattan-WRF version to include add electron density

observations/obs_converters

File / Directory Mods / Notes
buildall.csh New file, builds WRF-chem obs_converters
MOPITT_CO/ New directory, modified source files to be manhattan compliant if necessary, modified mkmf and path files to reflect Manhattan structures
AIRNOW/ New directory, modified source files to be manhattan compliant if necessary, modified mkmf and path files to reflect Manhattan structures
IASI_CO/ New directory, modified source files to be manhattan compliant if necessary, modified mkmf and path files to reflect Manhattan structures
IASI_O3/ New directory, modified source files to be manhattan compliant if necessary, modified mkmf and path files to reflect Manhattan structures
PANDA/ New directory, modified source files to be manhattan compliant if necessary, modified mkmf and path files to reflect Manhattan structures
MODIS Manhattan-WRF versions w/ mods: Added scripts/source, not vetted, not used in tutorial

RUN_SCRIPT MODS

The run script real_time_PANDA_RETR_RELEASE_TEST_NCAR_MANHATTAN.ksh drives the entire tutorial. Dates, configuration settings, etc. are set there, and this script calls a series of subscripts to run the entire tutorial. It was created from the original real_time_PANDA_RETR_RELEASE_TEST_CU.ksh script and modified to include job submissions for NCAR’s Cheyenne, and to be Manhattan compliant (e.g. paths reflect Manhattan directory structure, wrf_to_dart and dart_to_wrf sections removed).

It should be noted that there is an original NCAR script called real_time_PANDA_RETR_RELEASE_TEST_NCAR.ksh, but some of the settings in this script differed from the CU version. Since Arthur’s instructions were to use the CU script as a baseline, and the settings seem to reflect the purpose of his tutorial, the CU script was used at the baseline. It may be worth further exploring these differences between NCAR and CU versions in more detail in the future, or making sure the appropriate configuration is being set (would require input from Arthur).

Clone this wiki locally