Feature #1711, #1783, #1814, and #1835 Documentation - Multiple files…

…, PCPCombine constant init, Docker instructions, and MET links (#1967)

Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu>

docs/Users_Guide/getting_started.rst

@@ -377,3 +377,76 @@ the path to the METplus log file that was generated.
 * Review the :ref:`metplus_final.conf<metplus_final_conf>` file to see all
   of the settings that were used in the use case.
 
+
+.. _metplus-docker:
+
+METplus in Docker
+=================
+
+METplus is available on DockerHub. The METplus Docker image includes all of
+the MET executables from the corresponding METplus Coordinated Release and
+the METplus wrappers are pre-configured to use them.
+
+To pull the latest official release, run::
+
+    docker pull dtcenter/metplus:latest
+
+Tags for previous releases and development releases are also available.
+Refer to the list of
+`available tags <https://hub.docker.com/repository/docker/dtcenter/metplus/tags>`_
+on DockerHub.
+
+.. _docker-sample-input:
+
+Sample Input Data
+-----------------
+
+Sample input data for all of the use cases provided with the METplus wrappers
+are also available on DockerHub. These data are found in the
+dtcenter/metplus-data DockerHub repository and are named with the X.Y version
+of the corresponding METplus Coordinated Release and the name of the use case category
+separated by a dash, e.g. 4.1-data_assimilation or 4.0-met_tool_wrapper.
+A list of
+`available tags for input data <https://hub.docker.com/repository/docker/dtcenter/metplus-data/tags>`_
+can also be found on DockerHub.
+
+To make these data available in a METplus Docker container, first create a
+Docker data volume from the desired tag and give it a name with the *--name*
+argument::
+
+    docker create --name met_tool_wrapper dtcenter/metplus-data:4.1-met_tool_wrapper
+
+Then mount the data volume to the container using the *--volumes-from* argument
+to the *docker run* command::
+
+    docker run --rm -it --volumes-from met_tool_wrapper dtcenter/metplus:4.1.4 bash
+
+The input data will be available inside the container under
+/data/input/METplus_Data.
+
+Running METplus in Docker
+-------------------------
+
+The run_metplus.py script is in the user's path inside the container.
+The use case configuration files can be found under
+/metplus/METplus/parm/use_cases.
+The values for :ref:`sys_conf_met_install_dir`, :ref:`sys_conf_input_base`, and
+:ref:`sys_conf_output_base` are set to appropriate default values.
+A use case can be run by simply passing the use case configuration file to the
+run script::
+
+    run_metplus.py /metplus/METplus/parm/use_cases/met_tool_wrapper/Example/Example.conf
+
+Output from the use case will be written to /data/output by default.
+The value for :ref:`sys_conf_output_base` can be changed to write elsewhere::
+
+    run_metplus.py /metplus/METplus/parm/use_cases/met_tool_wrapper/Example/Example.conf config.OUTPUT_BASE=/data/output/my_data_dir
+
+If :ref:`docker-sample-input` is mounted to the container, then use cases from
+the corresponding category can be run::
+
+    run_metplus.py /metplus/METplus/parm/use_cases/met_tool_wrapper/GridStat/GridStat.conf
+
+Please note that use cases that have additional Python package dependencies
+may not run successfully unless those packages are installed inside the
+container or obtained elsewhere.

docs/Users_Guide/installation.rst

@@ -44,7 +44,7 @@ The following software is required to run METplus Wrappers:
 
 -  Python 3.8.6 or above
 
--  MET version 10.0.0 or above - 
+-  MET version 11.0.0 or above -
    For information on installing MET please see the `Software Installation/Getting Started <https://met.readthedocs.io/en/latest/Users_Guide/installation.html>`_
    section of the MET User's Guide. 
 

docs/Users_Guide/systemconfiguration.rst

@@ -760,10 +760,8 @@ then three times will be processed:
 2. Initialized on 2019-02-01 at 00Z / valid on 2019-02-01 at 06Z
 3. Initialized on 2019-02-01 at 00Z / valid on 2019-02-01 at 09Z
 
-The user can also define :term:`LEAD_SEQ` using a special notation for many
-forecast leads. The notation is **begin_end_incr(b,e,i)** where b = the
-first lead value, e = the last lead value (inclusive), and
-i = the increment between leads. For example::
+The user can also define :term:`LEAD_SEQ` using :ref:`begin_end_incr` for many
+forecast leads. For example::
 
   [config]
   LEAD_SEQ = begin_end_incr(0,12,3)
@@ -782,9 +780,8 @@ Any number of these groups can be defined by setting
 configuration variables LEAD_SEQ_1, LEAD_SEQ_2, ..., :term:`LEAD_SEQ_\<n\>`.
 The value can be defined with a
 comma-separated list of integers (currently only hours are supported here)
-or using the special begin_end_incr(b,e,i) notation described just
-above. Each :term:`LEAD_SEQ_\<n\>` must have a corresponding
-variable :term:`LEAD_SEQ_<n>_LABEL`. For example::
+or using :ref:`begin_end_incr`. Each :term:`LEAD_SEQ_\<n\>` must have a
+corresponding variable :term:`LEAD_SEQ_<n>_LABEL`. For example::
 
 
   [config]
@@ -891,8 +888,7 @@ Example 3::
 This will skip every 30th and 31st day **and** every 3rd month.
 
 
-**begin_end_incr(b,e,i)** syntax can be used to define a range of times to
-skip.
+:ref:`begin_end_incr` syntax can be used to define a range of times to skip.
 
 b = begin value, e = end value,
 
@@ -1640,10 +1636,80 @@ and will substitute the current day into the template, giving a filename of
 a file time of 20190131_23Z, so the filename will be 20190131.ext that is
 generated by the template.
 
-Using Windows to find Valid Files
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-The [FCST/OBS]_FILE_WINDOW_[BEGIN/END] configuration variables can be used
+Multiple Input Files
+^^^^^^^^^^^^^^^^^^^^
+
+If a tool supports reading multiple files for a given input, then there are
+a variety of ways to configure the METplus wrappers to read them.
+
+* :ref:`Wildcards <multiple-files-wildcard>`
+* :ref:`List of Templates <multiple-files-list>`
+* :ref:`File Windows <multiple-files-window>`
+
+.. _multiple-files-wildcard:
+
+Using Wildcards to find Multiple Files
+""""""""""""""""""""""""""""""""""""""
+
+Wildcard characters can be used in filename template variables.
+The \* character is used to match 1 or more characters and
+the \? character is used to match a single character.
+
+For example, if a directory /my/files contains the following files:
+
+* filename_AAA.nc
+* filename_ABA.nc
+* filename_BBB.nc
+* filename.nc
+
+The following template will match
+filename_AAA.nc, filename_ABA.nc and filename_BBB.nc, but not filename.nc::
+
+    INPUT_TEMPLATE = /my/files/filename_*.nc
+
+The following template will match filename_AAA.nc and filename_ABA.nc::
+
+    INPUT_TEMPLATE = /my/files/filename_A?A.nc
+
+
+.. _multiple-files-list:
+
+Using a List of Templates to find Multiple Files
+""""""""""""""""""""""""""""""""""""""""""""""""
+
+A comma-separated list of templates can be specified in a \_TEMPLATE variable.
+Each value in the list will be added to the corresponding \_DIR variable.
+
+For example, if a directory /my/files contains the following files:
+
+* filename_AAA.nc
+* filename_ABA.nc
+* filename_BBB.nc
+* filename.nc
+
+The following configuration will look for files
+/my/files/filename_AAA.nc and /my/files/filename_BBB.nc::
+
+    INPUT_DIR = /my/files
+    INPUT_TEMPLATE = filename_AAA.nc, filename_BBB.nc
+
+Lists of templates can be used with :ref:`wildcards <multiple-files-wildcard>`.
+The following configuration will find all 4 files in /my/files::
+
+    INPUT_DIR = /my/files
+    INPUT_TEMPLATE = filename.nc, filename_*.nc
+
+The :ref:`begin_end_incr` syntax can be used to generate lists of file paths.
+
+.. _multiple-files-window:
+
+Using File Windows to find Multiple Files
+"""""""""""""""""""""""""""""""""""""""""
+
+The :term:`FCST_FILE_WINDOW_BEGIN`, :term:`FCST_FILE_WINDOW_END`,
+:term:`OBS_FILE_WINDOW_BEGIN`, and :term:`OBS_FILE_WINDOW_END`
+configuration variables can be used
 if the time information in the input data does not exactly line up with the
 run time but the user still wants to process the data. The default value of the
 file window begin and end variables are both 0 seconds. If both values are
@@ -1665,12 +1731,11 @@ configuration::
 For a run time of 20190201_00Z, and a set of files in the input directory
 that looks like this:
 
-|   /my/grid_stat/input/obs/20190131/pre.20190131_22.ext
-|   /my/grid_stat/input/obs/20190131/pre.20190131_23.ext
-|   /my/grid_stat/input/obs/20190201/othertype.20190201_00.ext
-|   /my/grid_stat/input/obs/20190201/pre.20190201_01.ext
-|   /my/grid_stat/input/obs/20190201/pre.20190201_02.ext
-|
+* /my/grid_stat/input/obs/20190131/pre.20190131_22.ext
+* /my/grid_stat/input/obs/20190131/pre.20190131_23.ext
+* /my/grid_stat/input/obs/20190201/othertype.20190201_00.ext
+* /my/grid_stat/input/obs/20190201/pre.20190201_01.ext
+* /my/grid_stat/input/obs/20190201/pre.20190201_02.ext
 
 The following behavior can be expected for each file:
 
@@ -1690,12 +1755,11 @@ The following behavior can be expected for each file:
   5. The fifth file matches the template but it is a further distance away
      from the closest file (7200 seconds versus 3600 seconds) so it is ignored.
 
-Therefore, METplus Wrappers will use
-/my/grid_stat/input/obs/20190131/pre.20190131_23.ext as the input to
-grid_stat in this example.
+Therefore, /my/grid_stat/input/obs/20190131/pre.20190131_23.ext will be used
+as the input to grid_stat in this example.
 
-Wrapper Specific Windows
-^^^^^^^^^^^^^^^^^^^^^^^^
+
+**Wrapper Specific Windows**
 
 A user may need to specify a different window on a wrapper-by-wrapper basis.
 If this is the case, the user can override the file window values for each
@@ -1715,6 +1779,7 @@ EnsembleStat will use -0/+3600 for observation data.
 :term:`OBS_ENSEMBLE_STAT_FILE_WINDOW_BEGIN` was not set, so the
 EnsembleStat wrapper will use :term:`OBS_FILE_WINDOW_BEGIN`.
 
+
 .. _Runtime_Freq:
 
 Runtime Frequency
@@ -1914,6 +1979,41 @@ Files Processed::
     I2020101912_F006_V18
 
 
+.. _config-utilities:
+
+Config Utilities
+----------------
+
+.. _begin_end_incr:
+
+Begin End Increment (begin_end_incr)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In configuration variables that can accept a list of values, the
+Begin End Increment utility can be used to easily create a sequence of numbers
+without having to type out the entire list explicitly.
+This functionality is similar to the Python range() function except that it is
+inclusive, meaning that the end value is also included in the list.
+
+The notation is **begin_end_incr(b,e,i)** where b = the first lead value,
+e = the last lead value (inclusive), and i = the increment between values.
+
+begin_end_incr(0,6,2) will expand to a list of numbers from 0 to 6 by 2, or
+0, 2, 4, 6.
+
+An optional 4th argument can be provided to specify the zero padding.
+begin_end_incr(8,10,1,2) will expand to 08, 09, 10.
+
+If this syntax is found within a configuration variable, it will expand a
+string into a list with each number included. For example::
+
+    INPUT_TEMPLATE = ens01.nc, ens02.nc, ens03.nc, ens04.nc, ens05.nc, ens06.nc, ens07.nc, ens08.nc
+
+can be simplified as::
+
+    INPUT_TEMPLATE = ensbegin_end_incr(1,8,1,2).nc
+
+
 .. _metplus-control-met:
 
 How METplus controls MET configuration variables