Turn off two namelist parameters as MPAS V8+ no longer have reconstru…

…cture winds in lbc files. These options may need to be removed eventually.

CHANGELOG.rst

@@ -22,6 +22,11 @@ individual files.
 
 The changes are now listed with the most recent at the top.
 
+**May 16 2024 :: WRF v4. Tag v11.5.0**
+
+- WRF-DART and WRF-DART Tutorial updated to WRFv4. Note, not backwards compatible with WRFv3.9. 
+- local particle filter default value for pf_enkf_hybrid=.false. *contributed by Jon Poterjoy*
+
 **April 23 2024 :: Bug-fix: WRF hybrid vertical coordinate. Tag v11.4.1**
 
 - DART now detects whether WRF is using Hybrid Vertical Coordinate (HVC) introduced in WRFv3.9 or terrain following (TF) system.

assimilation_code/modules/assimilation/assim_tools_mod.pf.f90

@@ -165,7 +165,7 @@ module assim_tools_mod
 real(r8) :: pf_alpha                        = 0.30_r8
 integer  :: pf_kddm                         = 0
 logical  :: sampling_weighted_prior         = .true.
-logical  :: pf_enkf_hybrid                  = .true.
+logical  :: pf_enkf_hybrid                  = .false.
 real(r8) :: min_residual                    = 0.5_r8
 integer  :: pf_maxiter                      = 3
 real(r8) :: pf_kf_rtps_coeff                = 0.0_r8

conf.py

@@ -21,7 +21,7 @@
 author = 'Data Assimilation Research Section'
 
 # The full version, including alpha/beta/rc tags
-release = '11.4.1'
+release = '11.5.0'
 root_doc = 'index'
 
 # -- General configuration ---------------------------------------------------

guide/matlab-observation-space.rst

@@ -2,6 +2,8 @@
 MATLAB observation space diagnostics
 ####################################
 
+.. _configMatlab:
+
 Configuring MATLAB
 ==================
 

models/mpas_atm/README

@@ -4,54 +4,93 @@
 #
 # DART $Id$
 
-This directory contains the interface code and the shell scripts to
-run the MPAS-ATM atmospheric model with DART ensemble data assimilation.
+This directory contains the interface codes and the shell scripts to
+run the MPAS atmospheric model with DART ensemble data assimilation.
 
-The 'work' directory is where the executables are built.
-Run './quickbuild.sh' to compile all the executables.
-'./quickbuild.sh nompi' builds the serial versions only,
-'./quickbuild.sh' is the default and builds MPI (parallel) versions
-of the filter and any other executables that can run on multiple CPUs.
+The MPAS-DART interface is built on the MPAS model's native meshes, producing
+ensemble analysis in either quasi-uniform or variable-resolution meshes.
+Both regional and global simulations are supported with MPAS V7+, and 
+shell scripts are provided for cycling with both configurations.
+
+BUILDING the MPAS-A model
+==========================
+
+This README is provided as a brief introduction to the MPAS/DART framework.
+It does not provide details about the MPAS model, nor does it provide 
+building instructions for the model. 
+
+For information about downloading and building the MPAS model, please refer to the
+github directory (https://github.com/MPAS-Dev/MPAS-Model/releases) or the 
+tutorial website (https://www2.mmm.ucar.edu/projects/mpas/tutorial/v8.0/index.html).
+
+The Data Assimilation section in the MPAS documentation is also found at 
+http://mpas-dev.github.io.
+
+
+Code Layout
+============
+
+Within the mpas_atm directory, code is laid out as follows.
+DART/models/mpas_atm
+             ├── data: sample data and templates for MPAS and filter runs
+             ├── shell_scripts: shell scripts for cycling experiments
+             ├── tutorial: Hands-on tutorial with sample data
+             ├── matlab: matlab scripts for simple diagnostics
+             └── work: a place to build all MPAS/DART executables
+
+The 'data' directory contains small example MPAS files, namelist, and
+I/O stream files for an MPAS run.  
 
 The 'shell_scripts' directory contains a collection of scripts that 
-support the workflow for cycling.  The ensemble of MPAS-ATM models have 
+support the workflow for cycling.  The ensemble of the MPAS model has 
 to run to advance the time, then 'filter' runs to do data assimilation 
 on the ensemble of MPAS restart files, then the ensemble of MPAS models 
 advances again, then 'filter' runs, in a cycle.
+See shell_scripts/README for the cycling workflow.
+
+Compilation
+===================================
+The DART interface for MPAS can be compiled with various fortran compilers
+such as (but not limited to) gfortran, pgf90, and intel. It has been tested on a
+Mac and NSF NCAR Supercomputers (Derecho).
+
+Note:  While older versions of MPAS (prior to V7) requires the PIO (Parallel IO)
+       and pNetCDF (Parallel NetCDF) libraries, 
+       DART uses only the plain NetCDF libraries. If an altered NetCDF
+       library is required by the parallel versions, there may be incompatibilities
+       between the run-time requirements of DART and MPAS. Static linking of one or
+       the other executable, or swapping of modules between executions may be
+       necessary. That being said, all the input and output files are in NetCDF format
+       (except for obs sequence files).
+
+BUILDING the MPAS/DART interface
+===================================
+To build the MPAS/DART interface, go to DART/build_templates/ first and 
+choose the appropriate mkmf.template file for your system and fortran 
+compiler (ex. gfortran), make a link like below.
+
+DART/build_templates> ln -s mkmf.template.gfortran mkmf.template
+
+After that, come back to the 'work' directory here and
+run './quickbuild.sh' to compile all the executables based on mkmf.template.
+'./quickbuild.sh nompi' builds the serial versions only,
+'./quickbuild.sh' is the default and builds MPI (parallel) versions
+of the filter and any other executables that can run on multiple CPUs.
+
+=== Note ===
+1) If you want to compile in a single precision (r4), edit 
+DART/assimilation_code/modules/utilities/types_mod.f90, as below.
+
+!integer, parameter :: r8 = SELECTED_REAL_KIND(12)   ! 8 byte reals
+integer, parameter :: r8 = r4                      ! alias r8 to r4
+
+Without editing it, the default compilation will use double precision (r8).
 
-The 'data' directory contains a single small example MPAS-ATM restart
-file along with some stream files for an MPAS run.  See the README.txt
-for more details.
-
-Here are more details on how the information in the input and output
-files are handled in a cycling experiment.
-
-Very many thanks to Soyoung Ha (NCAR/MMM) and Ryan Torn (U Albany) for 
-developing and documenting these scripts.
-
-
-"filter_in.txt" : member1/restart.nc
-       |          member2/restart.nc
-       |
-       | as input_state_file_list in &filter_nml
-       V
-    <filter> - produces the EnKF analyses 
-       |
-       | as output_state_file_list in &filter_nml
-       V
-"filter_out.txt" : member1/analysis.nc
-       |           member2/analysis.nc
-       |
-       | as input_file_list in &update_mpas_states_nml
-       V
-<update_mpas_states> - updates analysis vector in MPAS restart files for all members
-       |
-       | as output_file_list in &update_mpas_states_nml
-       V
-"filter_in.txt" : member1/restart.nc
-       |          member2/restart.nc
-       | (Warning: restart.nc are overwritten for analysis fields.)
-       | same as input_state_file_list in &filter_nml 
-       V
-advance_model.csh ensemble_member_number
+2) To make cycling runs, you need to compile the MPAS model as indicated in
+   the MPAS tutorial shown above.
+3) If you need to create your own initial ensemble, you might need to install
+   WPS/ungrib.exe and run MPAS/init_atmosphere as well.
+   See shell_scripts/README.init_ensemble for details.
+4) MPAS files no longer need to be converted to DART formatted files,
+   they can be read in directly from a input file list!
 

models/mpas_atm/shell_scripts/README

@@ -1,45 +1,85 @@
-# How to create an initial mpas ensemble
+# MPAS/DART Cycling
+====================
 
-1) If you have access to an ensemble of grib files, then in the
-models/mpas_atm/shell_scripts directory are:
+This README describes a workflow at each analysis cycle with the input and output
+files specified for two ensemble members.
 
-driver_initial_ens.csh
-init_mpas_grib.csh
+For high-level data assimilation workflows in DART or the output files from filter 
+(e.g., files listed in `stages_to_write` in &filter_nml), see 
+https://docs.dart.ucar.edu/en/stable/guide/high-level-da-workflows.html.
 
-which will convert an ensemble of grib files to an ensemble
-of mpas files.
+Two more READMEs are provided in this directory:
+README.regional_mpas for regional MPAS modeling,
+README.init_ensemble for creating initial ensemble before cycling.
 
-2) If you do not have an ensemble of grib files but do have a single
-mpas file (either an initial or a restart file in netcdf), you can 
-create an ensemble with features built into the filter program itself.
 
-Set these parameters in input.nml.
+----------                         
+Workflow
+----------                         
 
-&filter_nml
-perturb_from_single_instance = .true.
+"filter_in.txt" : member1/restart.nc
+       |          member2/restart.nc
+       |
+       | as input_state_file_list in &filter_nml
+       V
+    <filter> - produces the EnKF analyses
+       |
+       | as output_state_file_list in &filter_nml
+       V
+"filter_out.txt" : member1/analysis.nc
+       |           member2/analysis.nc
+       |
+       | as input_file_list in &update_mpas_states_nml
+       V
+<update_mpas_states> - updates analysis vector in MPAS restart files for all members
+       |
+       | as output_file_list in &update_mpas_states_nml
+       V
+"filter_in.txt" : member1/restart.nc
+       |          member2/restart.nc
+       | same as input_state_file_list in &filter_nml
+       | (Warning: restart.nc are overwritten for analysis fields.)
+       V
+advance_model.csh ensemble_member_number
+(to run the MPAS model for each member with the member#/restart.nc until the next analysis time;
+ the resulting member#/restart.nc will be used as input to filter at the next analysis cycle)
 
-&model_nml
-model_perturbation_amplitude = 0.0001  	# set a small but non-zero value
 
-And make sure these parameters in &filter_nml:
+# Regional MPAS/DART
+---------------------
+For regional cycling, <update_bc> should be conducted right after <update_mpas_states>
+but before running advance_model.csh, with the namelist as below.
 
-input_state_file_list    = 'filter_in.txt'
-output_state_file_list   = 'filter_out.txt'
+&update_bc_nml
+  update_analysis_file_list           = 'filter_out.txt'
+  update_boundary_file_list           = 'boundary_inout.txt'
 
-have your single mpas filename in the filter_in.txt file,
-and have N filenames in the filter_out.txt file, one per
-line for each ensemble member. 
+Warning: lbc files listed in boundary_inout.txt will be overwritten (like restart.nc).
 
-(You can check models/mpas_atm/README for an example.)
 
-A single observation or a small set of observations in the obs sequence
-format can be set up in &filter_nml:
+-----
+Files
+-----                         
++----------------+-------------------------------------------------------------------+
+| filename       | purpose                                                           |
++================+======-============================================================+
+| input.nml      | to read the namelist - model_mod_nml and mpas_vars_nml            |
++----------------+-------------------------------------------------------------------+
+| restart.nc     | An MPAS restart file for each member, used as input (prior)       |
+|                | for filter and output (analysis) for the next model run           |
++----------------+-------------------------------------------------------------------+
+| analysis.nc    | time-variant analysis state (for each member) after assimilation  |
++----------------+-------------------------------------------------------------------+
+| mpas_init.nc   | init_template_filename for static info (ex. grid dimensions)      |
++----------------+-------------------------------------------------------------------+
 
-obs_sequence_in_name = 'obs_seq.1obs'
-
-Here the observation time in the obs sequence file should be matched with the 
-model time in the mpas file in 'filter_in.txt'.
-
-Run filter, which will perturb your single mpas file in filter_in.txt for 
-all the states listed as mpas_state_variables in &mpas_vars_nml in your input.nml,
-then write out N ensemble members as listed in filter_out.txt.  
+Note 1: File names in filter_out.txt are editable by users 
+        as long as they remain the same throughout the cycles.
+     2: restart.nc can be replaced by init.nc for the very first cycle.
+     3: For the model run in each member directory, ensure streams.atmosphere 
+        (for the model I/O streams) defines the output_interval for the restart files
+        as the cycling frequency (ex. output_interval="06:00:00" for 6-h cycling).
+     4: The restart file name defined in filter_in.txt for each member should match 
+        the restart filename_template defined in streams.atmosphere.
+     5: lbc files defined in boundary_inout.txt should match filename_template for "lbc_in" 
+        in streams.atmosphere.

models/mpas_atm/shell_scripts/README.init_ensemble

@@ -1,29 +1,75 @@
-# How to create a global initial ensemble from a single GFS analysis.
-# Soyoung Ha (MMM/NCAR) Feb-2020
+# How to create a global initial ensemble for cycling experiments.
+# Soyoung Ha (MMM/NCAR) Jun-2024
 
-1. Run WPS/ungrib.exe over the GFS analysis (ds084.1)
-   => GFS:YYYY-MM-DD_HH
+1) If you have access to an ensemble of grib files, then in the
+models/mpas_atm/shell_scripts directory are:
 
-2. Run MPAS/init_atmosphere (three times)
+driver_initial_ens.csh
+init_mpas_grib.csh
+
+which will convert an ensemble of grib files to an ensemble
+of mpas files.
+
+2) If you do not have an ensemble of grib files but do have a single
+global analysis file, you can create an ensemble with features built 
+into the filter program itself. The idea is to add tiny perturbations 
+to the analysis, lettting them grow with cycles or over forecast times.
+
+In this case, you need to have two more executables - WPS/ungrib.exe and MPAS/init_atmosphere.
+To download and install WPS, see https://github.com/wrf-model/WPS/.
+To build MPAS for initialization and for step 2 below,
+see https://www2.mmm.ucar.edu/projects/mpas/tutorial/v8.0/index.html.
+
+Let us start with a GFS analysis (https://rda.ucar.edu/datasets/ds084.1/).
+
+1. Run WPS/ungrib.exe over the GFS file in grib format.
+   (Use namelist.wps.global to edit start_date and end_date. No worries about the rest.)
+   => FILE:YYYY-MM-DD_HH
+
+2. Run MPAS/init_atmosphere (three times with config_init_case = 7, 
+   but changing streams.init_atmosphere for the corresponding input/output files.)
    => grid.nc, graph.info
    => static.nc
    => init.nc
 
-3. Run DART/models/mpas_atm/work/filter with 
+3. Run DART/models/mpas_atm/work/filter with input.nml edited as below.
+
+   &filter_nml
+   perturb_from_single_instance = .true.
    input_state_file_list    = 'filter_in.txt'
    output_state_file_list   = 'filter_out.txt'
-
-   (Check README and README.pert_ics on how to run it.)
-
-   filter_in.txt has a single line as
-   member1/init.nc 
+   obs_sequence_in_name     = 'obs_seq1.out'
+
+   &model_nml
+   model_perturbation_amplitude = 0.0001   # set a small but non-zero value
+
+   Note
+   ----
+   a) filter_in.txt has a single line as member1/init.nc 
    (which is copied from init.nc in step 2).
 
-   filter_out.txt looks like
+   b) filter_out.txt looks like
    member1/analysis.nc
    member2/analysis.nc
    ... (as many members as ens_size; one per line for each member)
 
+   c) DART/models/mpas_atm/data/obs_seq1.out can be used, but the observation time 
+   at the second last line (e.g., 0   151945) should be first matched with
+   the model time in the mpas file in 'filter_in.txt'.
+
+   To find the corresponding gregorian time for the obs_seq file, use 'advance_time' as below.
+   ncdump -v xtime member1/init.nc | tail -2 | head -1 | cut -d ";" -f1 | sed -e 's/"//g'  
+   # => 2024-06-13_12:00:00
+   echo 2024-06-13_12:00:00 0 -g | ./advance_time  # =>    154661   43200
+
+   Replace the second last line in obs_seq1.out with the new time (e.g., 43200   154661).
+   (Caution: The obs_seq file has seconds and days, not days and seconds.)
+
+   d) Leave the rest of the parameters in input.nml as they are, and run filter,
+   which will perturb your single mpas file in filter_in.txt for
+   all the states listed as mpas_state_variables in &mpas_vars_nml in your input.nml,
+   then write out N ensemble members as listed in filter_out.txt.
+
 4. Run DART/models/mpas_atm/work/update_mpas_states with an updated filter_in.txt.
    First, copy init.nc from step 2 to each member directory.
    Then edit filter_in.txt with all the members listed as below.
@@ -34,3 +80,15 @@
 
 5. Run MPAS/atmosphere_model up to the initial cycle time in each member directory.
    You might want to recenter the ensemble mean at the initial cycle before you get the cycling started.
+   Alternatively, you can skip this step and use initial ensemble (with tiny perturbations) generated
+   in step 4. In that case, you need to run many cycles until your ensemble spread gets saturated,
+   which might be costy. 
+   In the mesoscale applications, it is recommended to use the GFS analysis at least 7 days earlier than
+   the initial cycle time so that ensemble forecasts are run for 7 days (letting tiny perturbations grow).
+
+
+To create regional MPAS ensemble, the global ensemble produced above should be cut out for 
+the region of interest. To produce a regional file from a global MPAS file (for each member), 
+use creat_region in https://github.com/MPAS-Dev/MPAS-Limited-Area/. 
+Instructions can be found in Section 4.1 in the MPAS tutorial 
+(https://www2.mmm.ucar.edu/projects/mpas/tutorial/v8.0/index.html).