Per #1731, add ioda2nc documentation.

met/docs/Users_Guide/data_io.rst

@@ -10,13 +10,13 @@ Data must often be preprocessed prior to using it for verification. Several MET
 Input data formats
 __________________
 
-The MET package can handle gridded input data in one of four formats: GRIB version 1, GRIB version 2, NetCDF files following the Climate and Forecast (CF) conventions, and NetCDF files produced by the MET tools themselves. MET supports standard NCEP, USAF, UKMet Office and ECMWF grib tables along with custom, user-defined GRIB tables and the extended PDS including ensemble member metadata. See :numref:`Configuration File Details` for more information. Point observation files may be supplied in either PrepBUFR, ASCII, or MADIS format. Note that MET does not require the Unified Post-Processor to be used, but does require that the input GRIB data be on a standard, de-staggered grid on pressure or regular levels in the vertical. While the Grid-Stat, Wavelet-Stat, MODE, and MTD tools can be run on a gridded field at virtually any level, the Point-Stat tool can only be used to verify forecasts at the surface or on pressure or height levels. MET does not interpolate between native model vertical levels.
+The MET package can handle multiple gridded input data formats: GRIB version 1, GRIB version 2, and NetCDF files following the Climate and Forecast (CF) conventions, containing WRF output post-processed using wrf_interp, or produced by the MET tools themselves. MET supports standard NCEP, USAF, UKMet Office and ECMWF GRIB tables along with custom, user-defined GRIB tables and the extended PDS including ensemble member metadata. See :numref:`Configuration File Details` for more information. Point observation files may be supplied in either PrepBUFR, ASCII, or MADIS format. Note that MET does not require the Unified Post-Processor to be used, but does require that the input GRIB data be on a standard, de-staggered grid on pressure or regular levels in the vertical. While the Grid-Stat, Wavelet-Stat, MODE, and MTD tools can be run on a gridded field at virtually any level, the Point-Stat tool can only be used to verify forecasts at the surface or on pressure or height levels. MET does not interpolate between native model vertical levels.
 
 When comparing two gridded fields with the Grid-Stat, Wavelet-Stat, Ensemble-Stat, MODE, MTD, or Series-Analysis tools, the input model and observation datasets must be on the same grid. MET will regrid files according to user specified options. Alternatively, outside of MET, the copygb and wgrib2 utilities are recommended for re-gridding GRIB1 and GRIB2 files, respectively. To preserve characteristics of the observations, it is generally preferred to re-grid the model data to the observation grid, rather than vice versa.
 
-Input point observation files in PrepBUFR format are available through NCEP. The PrepBUFR observation files contain a wide variety of point-based observation types in a single file in a standard format. However, some users may wish to use observations not included in the standard PrepBUFR files. For this reason, prior to performing the verification step in the Point-Stat tool, the PrepBUFR file is reformatted with the PB2NC tool. In this step, the user can select various ways of stratifying the observation data spatially, temporally, and by type. The remaining observations are reformatted into an intermediate NetCDF file. The ASCII2NC tool may be used to convert ASCII point observations that are not available in the PrepBUFR files into this NetCDF format for use by the Point-Stat verification tool. Users with METAR or RAOB data from MADIS can convert these observations into NetCDF format with the MADIS2NC tool, then use them with the Point-Stat or Ensemble-Stat verification tools.
+Input point observation files in PrepBUFR format are available through NCEP. The PrepBUFR observation files contain a wide variety of point-based observation types in a single file in a standard format. However, some users may wish to use observations not included in the standard PrepBUFR files. For this reason, prior to performing the verification step in the Point-Stat tool, the PrepBUFR file is reformatted with the PB2NC tool. In this step, the user can select various ways of stratifying the observation data spatially, temporally, and by type. The remaining observations are reformatted into an intermediate NetCDF file. The ASCII2NC tool may be used to convert ASCII point observations that are not available in the PrepBUFR files into this common NetCDF point observation format. Several other MET tools, described below, are also provided to reformat point observations into this common NetCDF point observation format prior to passing them as input to the Point-Stat or Ensemble-Stat verification tools.
 
-Tropical cyclone forecasts and observations are typically provided in a specific ASCII format, in A Deck and B Deck files.
+Tropical cyclone forecasts and observations are typically provided in a specific ATCF (Automated Tropical Cyclone Forecasting) ASCII format, in A-deck, B-deck, and E-deck files.
 
 .. _Intermediate data formats:
 
@@ -55,32 +55,38 @@ The following is a summary of the input and output formats for each of the tools
 
 #. **PB2NC Tool**
 
-    * **Input**: One PrepBUFR point observation file and one configuration file.
+    * **Input**: PrepBUFR point observation file(s) and one configuration file.
 
     * **Output**: One NetCDF file containing the observations that have been retained.
 
 #. **ASCII2NC Tool**
 
-    * **Input**: One or more ASCII point observation file(s) that has (have) been formatted as expected, and optional configuration file.
+    * **Input**: ASCII point observation file(s) that has (have) been formatted as expected, and optional configuration file.
 
     * **Output**: One NetCDF file containing the reformatted observations.
 
 #. **MADIS2NC Tool**
 
-    * **Input**: One MADIS point observation file.
+    * **Input**: MADIS point observation file(s) in NetCDF format.
 
     * **Output**: One NetCDF file containing the reformatted observations.
 
 
 #. **LIDAR2NC Tool**
 
-    * **Input**: One CALIPSO satellite HDF file
+    * **Input**: One CALIPSO satellite HDF file.
+
+    * **Output**: One NetCDF file containing the reformatted observations.
+
+#. **IODA2NC Tool**
+
+    * **Input**: IODA observation file(s) in NetCDF format.
 
     * **Output**: One NetCDF file containing the reformatted observations.
 
 #. **Point2Grid Tool**
 
-    * **Input**: One NetCDF file containing point observation from the ASCII2NC, PB2NC, MADIS2NC, or LIDAR2NC tool.
+    * **Input**: One NetCDF file in the common point observation format.
 
     * **Output**: One NetCDF file containing a gridded representation of the point observations.
 
@@ -116,7 +122,7 @@ The following is a summary of the input and output formats for each of the tools
 
 #. **Point-Stat Tool**
 
-    * **Input**: One gridded model file, at least one point observation file in NetCDF format (as the output of the PB2NC, ASCII2NC, MADIS2NC, or LIDAR2NC tool), and one configuration file.
+    * **Input**: One gridded model file, at least one NetCDF file in the common point observation format, and one configuration file.
 
     * **Output**: One STAT file containing all of the requested line types and several ASCII files for each line type requested.
 
@@ -194,9 +200,9 @@ The following is a summary of the input and output formats for each of the tools
 
 #. **TC-Pairs Tool**
 
-    * **Input**: At least one A-deck and one B-deck ATCF format file containing output from a tropical cyclone tracker and one configuration file. The A-deck files contain forecast tracks while the B-deck files are typically the NHC Best Track Analysis but could also be any ATCF format reference.
+    * **Input**: At least one A-deck or E-deck file and one B-deck ATCF format file containing output from a tropical cyclone tracker and one configuration file. The A-deck files contain forecast tracks, the E-deck files contain forecast probabilities, and the B-deck files are typically the NHC Best Track Analysis but could also be any ATCF format reference.
 
-    * **Output**: ASCII output with the suffix .tcstat.
+    * **Output**: ASCII output with the suffix .tcst.
 
 #. **TC-Stat Tool**
 
@@ -208,7 +214,7 @@ The following is a summary of the input and output formats for each of the tools
 
     * **Input**: One or more Tropical Cyclone genesis format files, one or more verifying operational and BEST track files in ATCF format, and one configuration file.
 
-    * **Output**: One STAT file containing all of the requested line types and several ASCII files for each line type requested.
+    * **Output**: One STAT file containing all of the requested line types, several ASCII files for each line type requested, and one gridded NetCDF file containing counts of track points.
 
 #. **TC-RMW Tool**
 

met/docs/Users_Guide/overview.rst

@@ -57,7 +57,7 @@ The MODE (Method for Object-based Diagnostic Evaluation) tool also uses gridded
 
 The MODE-TD tool extends object-based analysis from two-dimensional forecasts and observations to include the time dimension. In addition to the two dimensional information provided by MODE, MODE-TD can be used to examine even more features including displacement in time, and duration and speed of moving areas of interest.
 
-The Grid-Diag tools produce multivariate probability density functions (PDFs) that may be used either for exploring the relationship between two fields, or for the computation of percentiles generated from the sample for use with percentile thresholding. The output from this tool requires post-processing by METplus or user-provided utilities.
+The Grid-Diag tool produces multivariate probability density functions (PDFs) that may be used either for exploring the relationship between two fields, or for the computation of percentiles generated from the sample for use with percentile thresholding. The output from this tool requires post-processing by METplus or user-provided utilities.
 
 The Wavelet-Stat tool decomposes two-dimensional forecasts and observations according to the Intensity-Scale verification technique described by :ref:`Casati et al. (2004) <Casati-2004>`. There are many types of spatial verification approaches and the Intensity-Scale technique belongs to the scale-decomposition (or scale-separation) verification approaches. The spatial scale components are obtained by applying a wavelet transformation to the forecast and observation fields. The resulting scale-decomposition measures error, bias and skill of the forecast on each spatial scale. Information is provided on the scale dependency of the error and skill, on the no-skill to skill transition scale, and on the ability of the forecast to reproduce the observed scale structure. The Wavelet-Stat tool is primarily used for precipitation fields. However, the tool can be applied to other variables, such as cloud fraction.
 

met/docs/Users_Guide/reformat_point.rst

@@ -599,13 +599,13 @@ MADIS2NC tool
 _____________
 
 
-This section describes how to run the MADIS2NC tool. The MADIS2NC tool is used to reformat `Meteorological Assimilation Data Ingest System (MADIS) <http://madis.noaa.gov>`_ point observations into the NetCDF format expected by the MET statistics tools. Since the MADIS2NC tool simply performs a reformatting step, no configuration file is needed. The MADIS2NC tool supports many of the MADIS data types, as listed in the usage statement below. Support for additional MADIS data types may be added in the future based on user feedback.
+This section describes how to run the MADIS2NC tool. The MADIS2NC tool is used to reformat `Meteorological Assimilation Data Ingest System (MADIS) <http://madis.noaa.gov>`_ point observations into the NetCDF format expected by the MET statistics tools. An optional configuration file controls the processing of the point observations. The MADIS2NC tool supports many of the MADIS data types, as listed in the usage statement below. Support for additional MADIS data types may be added in the future based on user feedback.
 
 
 madis2nc usage
 ~~~~~~~~~~~~~~
 
-The usage statement for MADIS2NC tool is shown below:
+The usage statement for the MADIS2NC tool is shown below:
 
 .. code-block:: none
 		
@@ -635,7 +635,7 @@ Required arguments for madis2nc
 1. The **madis_file** argument is one or more input MADIS point observation files to be processed.
 
 
-2. The **netcdf_file** argument is the NetCDF output file to be written.
+2. The **out_file** argument is the NetCDF output file to be written.
 
 
 3. The argument **-type str** is a type of MADIS observations (metar, raob, profiler, maritime, mesonet or acarsProfiles).
@@ -778,7 +778,6 @@ We will not give a detailed description of each CALIPSO data product that lidar2
 **Layer_Base** gives the elevation in meters above ground level of the cloud base for each cloud level at each observation location. Similarly, **Layer_Top** gives the elevation of the top of each cloud layer. Note that if there are multiple cloud layers at a particular location, then there will be more than one base (or top) given for that location. For convenience, **Min_Base** and **Max_Top** give, respectively, the base elevation for the bottom cloud layer, and the top elevation for the top cloud layer. For these data types, there will be only one value per observation location regardless of how many cloud layers there are at that location.
 
 
-
 .. _lidar2nc_grib_code_table:
 
 .. list-table:: lidar2nc GRIB codes and their meaning, units, and abbreviations
@@ -837,7 +836,145 @@ We will not give a detailed description of each CALIPSO data product that lidar2
     - Horizontal Averaging
     - NA
     - Horizontal_Averaging
+
+
+IODA2NC tool
+____________
+
+
+This section describes the IODA2NC tool which is used to reformat IODA (Interface for Observation Data Access) point observations from the `Joint Center for Satellite Data Assimilation (JCSDA) <http://jcsda.org>`_ into the NetCDF format expected by the MET statistics tools. An optional configuration file controls the processing of the point observations. The IODA2NC tool reads NetCDF point observation files created by the `IODA Converters <https://github.com/JCSDA-internal/ioda-converters>`_. Support for interfacing with data from IODA may be added in the future based on user feedback.
+
+
+ioda2nc usage
+~~~~~~~~~~~~~
+
+The usage statement for the IODA2NC tool is shown below:
+
+.. code-block:: none
+		
+  Usage: ioda2nc
+         ioda_file
+         netcdf_file
+         [-config config_file]
+         [-obs_var var]
+         [-iodafile ioda_file]
+         [-valid_beg time]
+         [-valid_end time]
+         [-nmsg n]
+         [-log file]
+         [-v level]
+         [-compress level]
+
+ioda2nc has required arguments and can also take optional ones.
+
+Required arguments for ioda2nc
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+1. The **ioda_file** argument is an input IODA NetCDF point observation file to be processed.
+
+2. The **netcdf_file** argument is the NetCDF output file to be written.
+
+Optional arguments for ioda2nc
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+3. The **-config config_file** is a IODA2NCConfig file to filter the point observations and define time summaries.
+
+4. The **-obs_var var_list** setting is a comma-separated list of variables to be saved from input the input file (by defaults, saves "all").
+
+5. The **-iodafile ioda_file** option specifies additional input IODA observation files to be processed.
+
+6. The **-valid_beg time** and **-valid_end time** options in YYYYMMDD[_HH[MMSS]] format overrides the retention time window from the configuration file.
+
+7. The  **-nmsg n** indicates the number of IODA records to process.
+
+8. The **-log** file option directs output and errors to the specified log file. All messages will be written to that file as well as standard out and error. Thus, users can save the messages without having to redirect the output on the command line. The default behavior is no log file.
+
+9. The **-v level** option indicates the desired level of verbosity. The value of “level” will override the default setting of 2. Setting the verbosity to 0 will make the tool run with no log messages, while increasing the verbosity above 1 will increase the amount of logging.
+
+10. The **-compress level** option indicates the desired level of compression (deflate level) for NetCDF variables. The valid level is between 0 and 9. The value of “level” will override the default setting of 0 from the configuration file or the environment variable MET_NC_COMPRESS. Setting the compression level to 0 will make no compression for the NetCDF output. Lower number is for fast compression and higher number is for better compression.
+
+An example of the ioda2nc calling sequence is shown below:
+
+.. code-block:: none
+
+    ioda2nc \
+    ioda.NC001007.2020031012.nc ioda2nc.2020031012.nc \
+    -config IODA2NCConfig -v 3 -lg run_ioda2nc.log
       
+In this example, the IODA2NC tool will reformat the data in the input ioda.NC001007.2020031012.nc file and write the output to a file named ioda2nc.2020031012.nc. The data to be processed is specified by IODA2NCConfig, log messages will be written to the ioda2nc.log file, and the verbosity level is three.
+
+
+ioda2nc configuration file
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The default configuration file for the IODA2NC tool named **IODA2NcConfig_default** can be found in the installed *share/met/config* directory. It is recommended that users make a copy of this file prior to modifying its contents.
+
+The IODA2NC configuration file is optional and only necessary when defining filtering the input observations or defining time summaries. The contents of the default IODA2NC configuration file are described below.
+
+__________________
+
+.. code-block:: none
+
+		obs_window = { beg  = -5400; end  = 5400; }
+		mask       = { grid = "";    poly = "";   }
+		tmp_dir    = "/tmp";
+		version    = "VN.N";
+
+The configuration options listed above are common to many MET tools and are described in :numref:`config_options`.
+
+_________________
+
+
+.. code-block:: none
+
+		message_type           = [];
+		message_type_group_map = [];
+		message_type_map       = [];
+		station_id             = [];
+      elevation_range        = { ... };
+      level_range            = { ... };
+      obs_var                = [];
+		quality_mark_thresh    = 0;
+		time_summary           = { ... }
+
+The configuration options listed above are supported by other point observation pre-processing tools and are described in :numref:`pb2nc configuration file`.
+
+_________________
+
+.. code-block:: none
+
+		obs_name_map = [];
+
+This entry is an array of dictionaries, each containing a **key** string and **val** string which define a mapping of input IODA variable names to output variable names. The default IODA map, obs_var_map, is appended to this map.
+
+_________________
+
+.. code-block:: none
+		
+		metadata_map = [
+		{ key = "message_type"; val = "msg_type"; },
+		{ key = "station_id";   val = "report_identifier"; },
+		{ key = "pressure";     val = "air_pressure,pressure"; },
+		{ key = "height";       val = "height,height_above_mean_sea_level"; },
+		{ key = "elevation";    val = ""; }
+		];
+
+This entry is an array of dictionaries, each containing a **key** string and **val** string which define a mapping of metadata for IODA data files.
+
+_________________
+
+.. code-block:: none
+
+		missing_thresh = [ <=-1e9, >=1e9, ==-9999 ];
+
+The **missing_thresh** option is an array of thresholds. Any data values which meet any of these thresholds are interpreted as being bad, or missing, data.
+
+
+ioda2nc output
+~~~~~~~~~~~~~~
+
+The NetCDF output of the IODA2NC tool is structured in the same way as the output of the PB2NC tool described in :numref:`pb2nc output`.
+
 
 Point2Grid tool
 _______________