Add Observations documents converted from markdown

README.rst

@@ -208,7 +208,15 @@ References
    :maxdepth: 2
    :caption: Observations
 
-   guide/obs-seq-file
+   guide/preprocess-program
+   guide/introduction-obs-seq-file
+   guide/detailed-structure-obs-seq
+   guide/creating-obs-seq-synthetic
+   guide/creating-obs-seq-real
+   guide/available-observation-converters
+   guide/manipulating-with-obs-sequence-tool
+   guide/difference-between-type-and-quantity
+   guide/adding-support-new-type
 
 .. toctree::
    :maxdepth: 2

guide/adding-support-new-type.rst

@@ -0,0 +1,5 @@
+Adding support for a new observation TYPE
+=========================================
+
+If you would like to add support for a new observation TYPE, see 
+:doc:`/observations/forward_operators/obs_def_mod` for detailed information.

guide/available-observation-converters.rst

@@ -0,0 +1,71 @@
+Available observation converter programs
+========================================
+
+The **DART/observations/obs_converters** directory contains a variety of
+converter programs to read various external formats and convert the observations
+into the format required by DART.
+
+Each directory has at least one converter:
+
+-  `AIRS <AIRS/AIRS.html>`__
+-  `Aviso+/CMEMS <AVISO/AVISO.html>`__
+-  `Ameriflux <Ameriflux/level4_to_obs.html>`__
+-  `COSMOS <COSMOS/COSMOS_to_obs.html>`__
+-  `DWL <DWL/dwl_to_obs.html>`__
+-  `GOES <GOES/README.md>`__
+-  `GPSPW <GPSPW/README>`__
+-  `GSI2DART <GSI2DART/README>`__
+-  `GTSPP <GTSPP/GTSPP.html>`__
+-  `MADIS <MADIS/MADIS.html>`__
+-  `MIDAS <MIDAS/MIDAS_to_obs.html>`__
+-  `MODIS <MODIS/MOD15A2_to_obs.htm>`__
+-  `NCEP (prepbufr -> ascii) <NCEP/prep_bufr/prep_bufr.html>`__
+-  `NCEP (ascii -> obs_seq) <NCEP/ascii_to_obs/create_real_obs.html>`__
+-  `ROMS <ROMS/ROMS.htm>`__
+-  `SSEC <SSEC/SSEC.html>`__
+-  `SST <SST/SST.html>`__
+-  `SSUSI <SSUSI/convert_f16_edr_dsk.html>`__
+-  `WOD <WOD/WOD.html>`__
+-  `cice <cice/cice_to_obs.html>`__
+-  `gnd_gps_vtec <gnd_gps_vtec/README>`__
+-  `GPS <gps/gps.html>`__
+-  `ok_mesonet <ok_mesonet/ok_mesonet.html>`__
+-  `QuikSCAT <quikscat/QuikSCAT.html>`__
+-  `Radar <radar/radar.html>`__
+-  `snow <snow/snow_to_obs.html>`__
+-  `Text <text/text_to_obs.html>`__
+-  `tpw <tpw/tpw.html>`__
+-  `Tropical Cyclones <tropical_cyclone/tc_to_obs.html>`__
+-  `Var (little-r) <var/littler_tf_dart.html>`__
+-  `Var (radar) <var/rad_3dvar_to_dart.html>`__
+
+There are also a couple utilities of note:
+
+-  `even_sphere <even_sphere/README>`__ - a utility for generating evenly-spaced
+   observation locations that can then be used in a perfect model experiment.
+-  `obs_error <obs_error/README>`__ - modules that specify observation errors
+   based on what is used by ECMWF and NCEP
+
+In addition the following external program produces DART observation sequence
+files:
+
+-  `Observation Processing And Wind Synthesis
+   (OPAWS) <http://code.google.com/p/opaws/>`__: OPAWS can process NCAR Dorade
+   (sweep) and NCAR EOL Foray (netCDF) radar data. It analyzes (grids) data in
+   either two-dimensions (on the conical surface of each sweep) or
+   three-dimensions (Cartesian). Analyses are output in netCDF, Vis5d, and/or
+   DART (Data Assimilation Research Testbed) formats.
+
+For generating synthetic observations, see the
+`create_obs_sequence <../../assimilation_code/programs/create_obs_sequence/create_obs_sequence.html>`__
+program documentation. You can also generate observation files based on text
+input. See the `text_to_obs <text/text_to_obs.html>`__ program documentation. Or
+for simulating a large complex observing system, you can use the DART library
+routines in a Fortran program to compute the observation information and have
+the DART routines write the output file.
+
+See the
+`perfect_model <../../assimilation_code/programs/perfect_model_obs/perfect_model_obs.html>`__
+program documentation on how to run a model with a set of observations that have
+only locations, types, and times, and have the forward operators compute the
+observation values.

guide/creating-obs-seq-synthetic.rst

@@ -0,0 +1,171 @@
+Creating an obs_seq file from synthetic observations
+====================================================
+
+There are several steps to create an observation sequence file, which follows
+directly from the modular nature of the DART programming philosophy.
+
+1. Decide what observations you want to investigate and edit the
+   ``input.nml&obs_kind_nml`` block.
+2. Build and run *preprocess* to create code that supports the observations you
+   want.
+3. Build and run *create_obs_sequence* to define the specifics about the
+   observation you want.
+4. Build and run *create_fixed_network_sequence* to replicate those specifics
+   through time.
+5. Build and run *perfect_model_obs* to create an observation consistent with
+   the model state and specified error distribution at the requested times and
+   locations.
+
+Example: generating observations for the Lorenz ’63 model.
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+1) There are no ‘real’ observations for the Lorenz ’63 model, so the appropriate
+namelist settings are:
+
+::
+
+   &obs_kind_nml
+       assimilate_these_obs_types = 'RAW_STATE_VARIABLE'  /
+
+   &preprocess_nml
+        input_obs_def_mod_file = '../../../observations/forward_operators/DEFAULT_obs_def_mod.F90'
+       output_obs_def_mod_file = '../../../observations/forward_operators/obs_def_mod.f90'
+       input_obs_kind_mod_file = '../../../assimilation_code/modules/observations/DEFAULT_obs_kind_mod.F90'
+      output_obs_kind_mod_file = '../../../assimilation_code/modules/observations/obs_kind_mod.f90'
+                   input_files = '../../../observations/forward_operators/obs_def_1d_state_mod.f90'
+     /
+
+2) Run *preprocess* in the normal fashion.
+
+3) *create_obs_sequence* creates an *observation set definition* (typically
+named ``set_def.out``), the time-independent part of an observation sequence. It
+may help to think of it as trying to define what sorts of observations will be
+taken at one ‘reading’ … you walk out to the box and take temperature, humidity,
+and wind observations all at the same time and place, for example. You can think
+of it as one page in an observer’s notebook, and only contains the *location,
+type,* and *observational error characteristics* (normally just the diagonal
+observational error variance) for a related set of observations. There are no
+actual observation values, nor are there any times associated with the
+definition. The program is interactive and queries the user for the information
+it needs. Begin by creating a minimal observation set definition in which each
+of the 3 state variables of L63 is directly observed with an observational error
+variance of 1.0 for each observation. To do this, use the following input
+sequence (the text including and after # is a comment and does not need to be
+entered):
+
+The following is a screenshot (much of the verbose logging has been left off for
+clarity), the user input looks *like this*.
+
+::
+
+      [unixprompt]$ ./create_obs_sequence
+       Starting program create_obs_sequence
+       Initializing the utilities module.
+       Trying to log to unit   10
+       Trying to open file dart_log.out
+
+       --------------------------------------
+       Starting ... at YYYY MM DD HH MM SS =
+                       2017  3 28 10 15 30
+       Program create_obs_sequence
+       --------------------------------------
+
+       set_nml_output Echo NML values to log file only
+       Trying to open namelist log dart_log.nml
+       ------------------------------------------------------
+
+
+       -------------- ASSIMILATE_THESE_OBS_TYPES --------------
+       RAW_STATE_VARIABLE
+       -------------- EVALUATE_THESE_OBS_TYPES --------------
+       ------------------------------------------------------
+
+       ---------- USE_PRECOMPUTED_FO_OBS_TYPES --------------
+       ------------------------------------------------------
+
+       Input upper bound on number of observations in sequence
+      4
+       Input number of copies of data (0 for just a definition)
+      0
+       Input number of quality control values per field (0 or greater)
+      0
+       input a -1 if there are no more obs
+      0
+            Input -1 * state variable index for identity observations
+            OR input the name of the observation type from table below:
+            OR input the integer index, BUT see documentation...
+              1 RAW_STATE_VARIABLE
+      -1
+       input time in days and seconds
+      0 0
+       Input error variance for this observation definition
+      1.0
+       input a -1 if there are no more obs
+      0
+
+       { this gets repeated ... until you tell it to stop ... }
+
+       input a -1 if there are no more obs
+      -1
+       Input filename for sequence (  set_def.out   usually works well)
+       set_def.out
+       write_obs_seq  opening formatted file set_def.out
+       write_obs_seq  closed file set_def.out
+
+Rest assured that if you requested to assimilate more realistic observation
+types, you will be queried for appropriate information by *create_obs_sequence*.
+Below is a table that explains all of the input you should need to supply for
+observations of the L63 model state.
+
+::
+
+   4            # upper bound on num of observations in sequence
+   0            # number of copies of data (0 for just a definition)
+   0            # number of quality control values per field (0 or greater)
+   0            # -1 to exit/end observation definitions
+
+   -1           # observe state variable 1
+   0   0        # time -- days, seconds
+   1.0          # observational variance
+   0            # -1 to exit/end observation definitions
+
+   -2           # observe state variable 2
+   0   0        # time -- days, seconds
+   1.0          # observational variance
+   0            # -1 to exit/end observation definitions
+
+   -3           # observe state variable 3
+   0   0        # time -- days, seconds
+   1.0          # observational variance
+   -1           # -1 to exit/end observation definitions
+
+   set_def.out  # Output file name
+
+4) *create_fixed_network_sequence* takes the observation set definition and
+repeats it in time, essentially making multiple pages in our notebook. Again,
+the program is interactive and queries the user for information. You should be
+able to simply follow the prompts. The table below represents the input needed
+for the L63 example:
+
+::
+
+   set_def.out    # Input observation set definition file          
+   1              # Regular spaced observation interval in time      
+   1000           # 1000 observation times
+   0, 43200       # First observation after 12 hours (0 days, 12 * 3600 seconds)
+   0, 43200       # Observations every 12 hours
+   obs_seq.in     # Output file for observation sequence definition
+
+5) *perfect_model_obs* advances the model from the state defined by the initial
+conditions file specified in the ``input.nml`` and ‘applies the forward
+operator’ to harvest observations to fill in the observation sequence specified
+in ``obs_seq.in``. The observation sequence finally has values for the
+observations and is saved in a file generally named *obs_seq.out*.
+*perfect_model_obs* is namelist-driven, as opposed to the previous two (whose
+input is a lot harder to specify in a namelist). Take a look at (and modify if
+you like) the ``input.nml&perfect_model_obs_nml`` section of the namelist.
+
+The End. Not only should you have an observation sequence file (usually
+``obs_seq.out``) , you also have a file containing the exact evolution of the
+model consistent with those observations - the true state:
+``perfect_output.nc``.

guide/dart-quality-control.rst

@@ -3,7 +3,7 @@ DART quality control field
 
 For failing observations, the DART quality control (QC) field may indicate the
 reason. To learn more about how to intepret the QC field, see
-:doc:`obs-seq-file`.
+:doc:`detailed-structure-obs-seq`.
 
 `How to locate the different values in an obs_seq.final file. <Observations.md#obs_seq_overview>`__
 

guide/detailed-structure-obs-seq.rst

@@ -0,0 +1,40 @@
+Detailed structure of an obs_seq file
+=====================================
+
+Since the configuration of model ensembles and the characteristics of
+assimilated observations are highly variable, observation sequence files are
+highly adjustable to accomodate these variations.
+
+There are many extensible parts of an observation sequence file. The following
+aspects of an observation sequence file can be adjusted:
+
+- the number of observation kinds contained in the file
+- whether the locations have one or more components
+- how many quality control values are available for each observation
+- where those quality control values come from
+- how many copies of each observation there are
+
+The following two diagrams demonstrate the structure of an ``obs_seq.out`` and
+an ``obs_seq.final`` file, respectively. 
+
+.. note::
+
+   These example files are from entirely separate experiments. They were
+   selected to demonstrate the flexibility of the observation sequence file
+   format.
+
+obs_seq.out structure
+---------------------
+
+|obs_seq_out|
+
+obs_seq.final structure
+-----------------------
+
+|obs_seq_final|
+
+.. |obs_seq_out| image:: ./images/science_nuggets/obs_seq_out_diagram.png
+   :width: 100%
+
+.. |obs_seq_final| image:: ./images/science_nuggets/obs_seq_final_diagram.png
+   :width: 100%

guide/difference-between-type-and-quantity.rst

@@ -0,0 +1,37 @@
+The difference between observation TYPE and QUANTITY
+====================================================
+
+Since DART is designed to assimilate data from any data source into any model,
+the assimilation algorithms need a way to define how observational data sources
+relate to model state variables.
+
+DART does this by defining a single generic observation QUANTITY, such as zonal
+wind, and mapping many specific observation TYPEs, corresponding to source 
+observations, to the single QUANTITY.
+
+For example, QuikSCAT and radiosondes are both capable of measuring zonal wind.
+DART defines two observation TYPEs:
+
+- ``QKSWND_U_WIND_COMPONENT`` for the QuikSCAT observations of zonal wind
+- ``RADIOSONDE_U_WIND_COMPONENT`` for the radiosonde observations of zonal wind
+
+and relates both of these TYPES to a single QUANTITY: QTY_U_WIND_COMPONENT.
+
+Thus TYPE and QUANTITY have a many to one relationship. This distinction
+enables you to evaluate observation platforms independently of one another.
+
+The forward observation operators are implemented based on observation
+QUANTITY. When requested, the model generates a QTY_U_WIND_COMPONENT, it
+doesn't need to know that it will be compared to an observation from QuikSCAT
+or one from a radiosonde.
+
+.. tip::
+
+   It is usually scientifically very interesting to be able to compare the
+   assimilations one TYPE of observation verus another. An observation
+   sequence file can have many types of observations. DART has the capability
+   to assimilate (or evaluate) any combination of observation types without
+   getting bogged down in dataset management. The same observation sequence can
+   be used for experiments that include or exclude certain observation types.
+   This procedure can ensure that you are actually performing the experiment
+   that you think you are performing.

guide/examining-obs-seq-final.rst

@@ -17,7 +17,7 @@ If there are no changes in the model state after assimilation, then examine the
     
    Then rerun ``filter`` to regenerate an ``obs_seq.final`` file in ASCII. For 
    an explanation of the contents of your ``obs_seq.final`` file, see
-   :doc:`obs-seq-file`.
+   :doc:`detailed-structure-obs-seq`.
 
 2. If you are using many observations, run the ``obs_diag`` program appropriate
    for your model. The :doc:`matlab-observation-space` will help to summarize